[Discuss] 'C' string tokenizer for those who hate strtok

[David Bronaugh]

pw wrote:
> Brian Quinlan wrote:
>> I'm not actually sure that I agree with Peter's point that his code 
>> is somehow "safer" but I think that this discussion is raising some 
>> interesting issues.
>>
>> One is the trade-off between correctness/reliability and performance. 
>> By choosing C as your implementation language, I think that you've 
>> already taking a large step in deciding that you favor performance. 
>> Usually, you would be better to write the bulk of your language in a 
>> higher-level language and then implement the performance critical 
>> sections in C. Hopefully those sections would be small enough that 
>> you could carefully test and scrutinize them.
>>
>> You are also losing a lot in terms of programmer productivity, long 
>> term maintenance costs, etc. by using C.
>>
>> <ding> <ding> Round 3...
>>
>> Cheers,
>> Brian
>
> Actually the initial point was that this linked list approach was an 
> improvement over strtok, which it is. The code provides better 
> functionality, handles token storage dynamically
> (but that could also be improved),
> and allows repeated iteration which strtok does not.
>
> I'm not sure about productivity. There are some pretty nonproductive
> high level programs that require a great deal of specialized support.
> If the program is done properly in the first place there shouldn't be
> any long term maintenance costs.(unless it's programmed to require a 
> vampyric service contract)
>
> The point of writing better code in any language is to build libraries 
> of portable, useful and easily understandable (documented) routines.
> The reason python or any other language exists is because someone(s) sat
> down and coded that language with safety and robustness in mind.
>
For amusement I rewrote your main.c file... it was amusing.

http://bronaugh.linuxboxen.org/main.c

The major changes: I removed all the silly copying, I made the list 
singly linked, I removed the O(n) seeks in the string, and I think I 
trimmed about 100 lines off the source code. It's also typesafe now (no 
void*) and I added a few checks. I also removed code duplication.

Oh yeah, and for giggles I made the 'free' function recursive.

David