Go to
and download the plain text version of "War and Peace", then use this to make a fulltext index based on syllables instead of words:
Code:
#Define ccBaseDir 'C:\my\fulltextindex\'
t0 = Seconds()
lcWarAndPeace = Filetostr("C:\my\fulltextindex\2600.txt")
* double line feeds (CRLF) end a pragraph, sorten each end-of-paragraph to CRCR
lcWarAndPeace = Strtran(lcWarAndPeace ,Chr(13)+Chr(10)+Chr(13)+Chr(10),Chr(13)+Chr(13))
* remaining single CRLFs are not wanted, we want running text - no CRLFs, only to end paragraphs
lcWarAndPeace = Strtran(lcWarAndPeace ,Chr(13)+Chr(10),'')
* NOw change CRCR to single CR, this is used with ALINES as parsechar
lcWarAndPeace = Strtran(lcWarAndPeace ,Chr(13)+Chr(13),Chr(13))
? 'changing text to running text', Seconds()-t0
* create an array of paragraphs, finally a table tabWarAndPeace.dbf
t0=Seconds()
lnParagraphs = Alines(laParagraphs,lcWarAndPeace ,1+2+4,Chr(13))
Close Databases All
Close Tables All
Create Cursor curWarAndPeace (iid I Autoinc, mParagraph M)
For Each lcParagraph In laParagraphs FoxObject
Insert Into curWarAndPeace (mParagraph) Values (lcParagraph)
Endfor
? 'inserting paragraphs into a cursor', Seconds()-t0
t0=Seconds()
Create Database (ccBaseDir+"WarAndPeaceIndex.dbc")
Copy To (ccBaseDir+"tabWarAndPeace.dbf") Database WarAndPeaceIndex
? 'write cursor to table', Seconds()-t0
t0=Seconds()
Create Table (ccBaseDir+"tabResults") (SyllableID I, ParagraphID I)
Index On SyllableID Tag xfSyllabID
Index On ParagraphID Tag xfParagrID Additive
Index On BinToC(SyllableID)+BinToC(ParagraphID) Tag xKeyPair Collate "MACHINE" Additive
Create Table (ccBaseDir+"tabSyllables") (SyllableID I Autoinc, Primary Key SyllableID Tag xpSyllabID, cSyllable C(20))
Index On Lower(cSyllable) Tag xSyllable Additive
* minimum syllable length (later also minimum search term length
#Define cnMinLen 3
* maximum syllable length
#Define cnMaxLen 20
Local lnStart, lnLen, lcSyllable
Select 0
Use tabWarAndPeace In 0
Scan
For lnStart = 1 To Len(mParagraph)-cnMinLen+1
For lnLen = cnMinLen To cnMaxLen
lcSyllable = Lower(Alltrim(Substr(mParagraph,lnStart,lnLen)))
If Len(lcSyllable)<cnMinLen Or ' ' $ lcSyllable
Loop
Endif
If Not Seek(lcSyllable,"tabSyllables","xSyllable")
Insert Into tabSyllables (cSyllable) Values (lcSyllable)
Endif
If Not Seek(BinToC(tabSyllables.SyllableID)+BinToC(tabWarAndPeace.iid),"tabResults","xKeyPair")
Insert Into tabResults Values (tabSyllables.SyllableID, tabWarAndPeace.iid)
Endif
Endfor
Endfor
Endscan
* Flush to disc:
FLUSH
Close Tables All
? 'create index on syllables', Seconds()-t0
t0=Seconds()
Select tabWarAndPeace.mParagraph;
From tabSyllables;
inner Join tabResults On tabResults.SyllableID = tabSyllables.SyllableID;
inner Join tabWarAndPeace On tabWarAndPeace.iID = tabResults.ParagraphID;
Where Lower(tabSyllables.cSyllable) = "fox";
into Cursor curFoxInWarAndPeace nofilter
? 'find "fox" with syllables index', Seconds()-t0
t0=Seconds()
Select tabWarAndPeace.mParagraph;
From tabWarAndPeace;
Where "fox" $ Lower(mParagraph);
into Cursor curFoxInWarAndPeace2 nofilter
? 'find "fox" without syllables index', Seconds()-t0
Heres some statistics:
Original plain text: 3212 KB
Read into Table of Paragraphs: 3647 KB
Syllable table: ~33 MB
Results Table (keypairs connecting paragraphs and syllables): 133 MB
The search result with using $ takes about 2 to 4 times as long as searching for a syllable.
There are 16 paragraphs in War And Peace containing "fox":
...The weather was already growing wintry and morning frosts congealedan earth saturated by autumn rains. The verdure had thickened and itsbright green stood out sharply against the brownish strips of winter ryetrodden down by the cattle, and against the pale-yellow stubble of thespring buckwheat. The wooded ravines and the copses, which at the end ofAugust had still been green islands amid black fields and stubble, hadbecome golden and bright-red islands amid the green winter rye. Thehares had already half changed their summer coats, the fox cubs werebeginning to scatter, and the young wolves were bigger than dogs. It wasthe best time of the year for the chase. The hounds of that ardent youngsportsman Rostov had not merely reached hard winter condition, but wereso jaded that at a meeting of the huntsmen it was decided to give thema three days' rest and then, on the sixteenth of September, to go ona distant expedition, starting from the oak grove where there was anundisturbed litter of wolf cubs...
..."A perfect picture! How he chased a fox out of the rank grass by theZavarzinsk thicket the other day! Leaped a fearful place; what a sightwhen they rushed from the covert... the horse worth a thousand rublesand the rider beyond all price! Yes, one would have to search far tofind another as smart."...
...Facing him lay a field of winter rye, there his own huntsman stood alonein a hollow behind a hazel bush. The hounds had scarcely been loosedbefore Nicholas heard one he knew, Voltorn, giving tongue at intervals;other hounds joined in, now pausing and now again giving tongue. Amoment later he heard a cry from the wooded ravine that a fox had beenfound, and the whole pack, joining together, rushed along the ravinetoward the ryefield and away from Nicholas...
...He saw the whips in their red caps galloping along the edge of theravine, he even saw the hounds, and was expecting a fox to show itselfat any moment on the ryefield opposite...
...The huntsman standing in the hollow moved and loosed his borzois, andNicholas saw a queer, short-legged red fox with a fine brush going hardacross the field. The borzois bore down on it.... Now they drew closeto the fox which began to dodge between the field in sharper and sharpercurves, trailing its brush, when suddenly a strange white borzoi dashedin followed by a black one, and everything was in confusion; the borzoisformed a star-shaped figure, scarcely swaying their bodies and withtails turned away from the center of the group. Two huntsmen galloped upto the dogs; one in a red cap, the other, a stranger, in a green coat...
...The huntsmen got the fox, but stayed there a long time without strappingit to the saddle. Their horses, bridled and with high saddles, stoodnear them and there too the dogs were lying. The huntsmen waved theirarms and did something to the fox. Then from that spot came the sound ofa horn, with the signal agreed on in case of a fight...
...Nicholas dismounted, and with Natasha and Petya, who had ridden up,stopped near the hounds, waiting to see how the matter would end. Out ofthe bushes came the huntsman who had been fighting and rode towardhis young master, with the fox tied to his crupper. While still at adistance he took off his cap and tried to speak respectfully, but he waspale and breathless and his face was angry. One of his eyes was black,but he probably was not even aware of it...
..."A likely thing, killing a fox our dogs had hunted! And it was my graybitch that caught it! Go to law, indeed!... He snatches at the fox! Igave him one with the fox. Here it is on my saddle! Do you want a tasteof this?..." said the huntsman, pointing to his dagger and probablyimagining himself still speaking to his foe...
..The facts were that Ilagin, with whom the Rostovs had a quarrel and wereat law, hunted over places that belonged by custom to the Rostovs, andhad now, as if purposely, sent his men to the very woods the Rostovswere hunting and let his man snatch a fox their dogs had chased...
...Instead of an enemy, Nicholas found in Ilagin a stately and courteousgentleman who was particularly anxious to make the young count'sacquaintance. Having ridden up to Nicholas, Ilagin raised his beavercap and said he much regretted what had occurred and would have the manpunished who had allowed himself to seize a fox hunted by someone else'sborzois. He hoped to become better acquainted with the count and invitedhim to draw his covert...
...The serfs all dispersed. "Uncle" lifted Natasha off her horse and takingher hand led her up the rickety wooden steps of the porch. The house,with its bare, unplastered log walls, was not overclean--it did not seemthat those living in it aimed at keeping it spotless--but neither wasit noticeably neglected. In the entry there was a smell of fresh apples,and wolf and fox skins hung about...
...The valet brought a woman's fox-lined cloak...
..."Now then, you foxes!" said another, laughing at some militiamen who,stooping low, entered the battery to carry away the wounded man...
..."Where is he?" he inquired. And as he spoke he saw a young man cominground the corner of the house between two dragoons. He had a long thinneck, and his head, that had been half shaved, was again covered byshort hair. This young man was dressed in a threadbare blue cloth coatlined with fox fur, that had once been smart, and dirty hempen convicttrousers, over which were pulled his thin, dirty, trodden-down boots.On his thin, weak legs were heavy chains which hampered his irresolutemovements...
...Half an hour later he was driving with his fast horses across theSokolniki field, no longer thinking of what had occurred but consideringwhat was to come. He was driving to the Yauza bridge where he had heardthat Kutuzov was. Count Rostopchin was mentally preparing the angry andstinging reproaches he meant to address to Kutuzov for his deception. Hewould make that foxy old courtier feel that the responsibility for allthe calamities that would follow the abandonment of the city and theruin of Russia (as Rostopchin regarded it) would fall upon his dotingold head. Planning beforehand what he would say to Kutuzov, Rostopchinturned angrily in his caleche and gazed sternly from side to side...
...Pierre rushed to the wing, but the heat was so great that heinvoluntarily passed round in a curve and came upon the large housethat was as yet burning only at one end, just below the roof, and aroundwhich swarmed a crowd of Frenchmen. At first Pierre did not realizewhat these men, who were dragging something out, were about; but seeingbefore him a Frenchman hitting a peasant with a blunt saber and tryingto take from him a fox-fur coat, he vaguely understood that looting wasgoing on there, but he had no time to dwell on that idea...
Bye, Olaf.
