Commit 0d551497 authored by dagal's avatar dagal

Built CTR over EMT

parent b0a3e012
WARNING: 8 Unused stops
SKIPPED: 309937
ERRORS: 0
LENGTHS: Counter({2: 3846156, 4: 3846153, 6: 1923076, 8: 384615})
CHANGES: Counter({0: 3846156, 1: 3846153, 2: 1923076, 3: 384615})
0 0.3846156
1 0.3846153
2 0.1923076
3 0.0384615
real 206m16.782s
user 204m29.078s
sys 0m46.375s
\ No newline at end of file
......@@ -2,5 +2,6 @@
#zcat ./texts/porto.txt.gz | ../BUILDALLwcsa stdin ./indexes/porto_baseline "sPsi=512; nsHuff=16;psiSF=1"
#cat ./texts/madrid_ex.txt | ../BUILDALLwcsa stdin ./indexes/madrid_baseline "sPsi=512; nsHuff=16;psiSF=1"
cat ./texts/madrid_lines.zst | zstd -d | ../BUILDALLwcsa stdin ./indexes/madrid_lines "sPsi=512; nsHuff=16;psiSF=1; bTimes=RRR128;bLines=RRR128"
#cat ./texts/madrid_lines.zst | zstd -d | ../BUILDALLwcsa stdin ./indexes/madrid_lines "sPsi=512; nsHuff=16;psiSF=1; bTimes=RRR128;bLines=RRR128"
zcat ./texts/emt_trips.zip | ../BUILDALLwcsa stdin ./indexes/madrid_emt "sPsi=512; nsHuff=16;psiSF=1; bTimes=RRR128;bLines=RRR128"
#!/bin/bash
cat madrid_trips.zst | zstd -d | python3 lineStops.py > texts/lineStops.txt
cat madrid_trips.zst | zstd -d | python3 tiempoMedio.py texts/lineStops.txt > texts/avgTimes.txt
cat madrid_trips.zst | zstd -d | python3 tiempoInicial.py > texts/initialTimes.txt
zcat texts/emt_journeys.zip | python3 lineStops.py > texts/lineStops.txt
zcat texts/emt_journeys.zip | python3 tiempoMedio.py texts/lineStops.txt > texts/avgTimes.txt
zcat texts/emt_journeys.zip | python3 tiempoInicial.py > texts/initialTimes.txt
cat texts/lineStops.txt | python3 stopLines.py > texts/stopLines.txt
File mode changed from 100644 to 100755
out file set to : ./indexes/madrid_emt
Read weeks=0
Processing 388.5%
UNSORTED: 1281 > 4242 (record j=3)
INPUT RECORDS ARE #not# SORTED INCREASINGLY
UNSORTED: 2435 > 3538 (record j=4)
SORTED RECORDS ARE #not# SORTED INCREASINGLY
10000001 trajectories read, max nodes = 4697, max-time = 5760
[0 1 2405 340 211 ]
[0 1 2435 211 ]
[0 1 2435 211 ]
[0 1 3538 4058 211 ]
[0 1 2435 309 ]
[0 1 2435 309 ]
[0 1 3536 3096 1001 329 ]
[0 1 2436 696 338 ]
[0 1 2731 2899 338 ]
[0 1 2412 2899 338 ]
...
...
[0 2303 2345 2343 4557 ]
[0 2303 976 2343 4557 ]
[0 2303 978 3778 194 4593 ]
[0 2303 1173 2192 4596 ]
[0 2303 976 198 437 4637 ]
[0 2303 413 1719 4657 ]
[0 2303 3122 2052 4666 ]
[0 2304 1513 2535 3 ]
[0 2304 1513 7 3 ]
[0 2304 1515 5 3 ]
...
...
[0 4696 4675 ]
[0 4696 4675 ]
[0 4696 4675 ]
[0 4696 4675 ]
[0 4696 4675 ]
[0 4696 4675 ]
[0 4696 4675 ]
[0 4696 1987 3975 4694 ]
[0 4696 3245 3975 4694 ]
[0 ]
parameters: "sPsi=512; nsHuff=16;psiSF=1; bTimes=RRR128;bLines=RRR128"
Number of nodes = 4697
Number of times = 5761
real nodes 4689
map_size 4697 Number of lines = 420
map and unmap vocabulary arrays created sucessfully
**** CREATING CSA-bottom-layer *****
parameters for iCSA: samplePsi=512
: nsHuff=16384, psiSearchFactor = 1 --> jump = 512
*BUILDING THE SUFFIX ARRAY over 38846151 integers... (with sais)
...... ended.
Creating compressed Psi...
Creating compressed Psi HUFFMAN RLE...
MALLOC FOR 16384
MALLOC FOR 38846151
[3] diffsHT.total = 202703264 bits
[3]streamSize = 240692066 , index = 16384
psi: pointersize = 28 bits, sampleSize=26 bits
espacio para Sample-values-psi = 246584 bytes
espacio para Sample-values-psi** = 246584 bytes
espacio para Sample-pointers-psi = 265552 bytes
espacio para stream-psi = 30086512 bytes
@@@@@@@@@ psi samaplePeriod= 512, ns=16384
@@@@@@@@@ psi size= [samples = 246584] bytes
@@@@@@@@@ psi size= [pointers = 265552] bytes
@@@@@@@@@ psi size= [totalsize diffsHt.total = 202703264] bits
@@@@@@@@@ psi size= [streamsize+largevalues =30086512] bytes
@@@@@@@@@ psi size= [sizeHuff tree = 65584] bytes
**** [iCSA built on 38846151 integers. Size = 35530698 bytes... RAM
Test MAP/UNMAP (compressDictionary RRR) passed *OK*,
Building WM Indices...
Done.
Saving structures to disk: ./indexes/madrid_emt.*Index saved !!
Size of int index: 35533748 bytes, 7.318 bps (56.29% compression)
Size of lines index: 7354700 bytes, 1.515 bps (16.83% compression)
Size of times index: 61981664 bytes, 12.765 bps (98.19% compression)
Size of lineStops: 46956 bytes
Size of stopLines: 59372 bytes
Size of avgTimes: 25158 bytes
Size of initialTimes: 251160 bytes, 2.097 bps (6.55% compression)
Index occupied 105252758 bytes
Size of int index: 35533748 bytes, 7.318 bps (56.29% compression)
Size of lines index: 7354700 bytes, 1.515 bps (16.83% compression)
Size of times index: 61981664 bytes, 12.765 bps (98.19% compression)
Size of lineStops: 46956 bytes
Size of stopLines: 59372 bytes
Size of avgTimes: 25158 bytes
Size of initialTimes: 251160 bytes, 2.097 bps (6.55% compression)
[destroying index] ...Freed 105252758 bytes... RAM
[destroying iCSA: compressed PSI structure] ...Freed 30664360 bytes... RAM
[destroying iCSA: D vector] ...Freed 4855772 bytes... RAM
**** [the whole iCSA ocuppied ... 35530698 bytes... RAM
**** iCSA size = 35530698 bytes
## Building time (**parsing into integers + present_layer: 113.484 secs
File mode changed from 100644 to 100755
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment