How to get the token type from a CFStringTokenizer in Cocoa?

The CFStringTokenizer documentation has two conflicting statements in CFStringTokenizerAdvanceToNextToken():

CFStringTokenizerAdvanceToNextToken

...

Return Value

The type of the token if the tokenizer succeeded in finding a token and setting it as current token. Returns kCFStringTokenizerTokenNone if the tokenizer failed to find a token. For possible values, see “Token Types.”

...

If a token is found, it is set as the current token and the function returns true; otherwise the current token is invalidates and the function returns false.

The first paragraph (returning a token type) is what I'd like to see: it lets you, for example, check if a token is made up entirely of non-alphanumeric characters. However, the second paragraph (returning true or false) seems to be what is actually happening.

Any ideas why that would be, or how to work around it?

From stackoverflow qwzybug

• The header comment doesn't mention returning true or false, and when the header and the online docs disagree it's often the header that is correct.

  In a simple test I'm able to get return values other than 0 and 1, so the problem you are seeing may be more specific; can you post sample code that's failing?

  From smorgan

• It's possible they meant “true” and “false” in the broader sense of “non-zero” and “zero”. If it finds a token, the function returns kCFStringTokenizerTokenNormal (which is 1) or some combination of the masks (either way, non-zero/“true”). If it doesn't, the function returns kCFStringTokenizerTokenNone (which is zero/“false”).

  It's certainly vague language, though, so please file a documentation bug on the Apple Bug Reporter.

  From Peter Hosey