26 Jul 12 5:23
The simplest version :
CODE --> (Ba|K)Sh
while IFS=';' read -r begin end; do
[[ -f "$cache/$begin" ]] || dosomething "$begin" > "$cache/$begin"
echo "$(< "$cache/$begin" );$end"
done < file1.txt
This creates separate file for each code. Fast to write, fast to read, may be slow to search, but this probably depends on the used filesystem too.
Regarding that search slowness, I would just start the script, wait until there are a few thousand files in the cache directory, then do a [[ -f '/tmp/dosomething.cache/4567' ]]
( or any other code ) from the command line and see whether it takes whole seconds. If yes, tell us. Then we will look for other storage tricks ( for example separate subdirectories based on the first character ) or alternatives ( for example SQLite database ).
One thing to note :
Quote (man bash)
It's too big and too slow.
If you have Ksh, use that instead. ( On Linux you will probably find the public domain ( pdksh
) or MirOS ( mksh
) implementation. They are also faster. )
If you have Dash, use that instead. But Dash has only what POSIX specifies, so the above code will need minor rewrite.