IDNStudy.com, ang perpektong platform para magtanong at makakuha ng maaasahang mga sagot. Magtanong at makatanggap ng eksaktong sagot mula sa aming mga bihasang miyembro ng komunidad.
Answer:
Some time ago, I got inspired to build an app for solving specific kinds of math problems. I discovered I had to parse the expression into an abstract syntax tree, so I decided to build a prototype in Javascript. While working on the parser, I realized the tokenizer had to be built first. I’ll walk you through how to do one yourself. (Warning: its easier than it looks at first.)
What is a Tokenizer?
A tokenizer is a program that breaks up an expression into units called tokens. For instance, if we have an expression like “I’m a big fat developer”, we could tokenize it in different ways, such as:
Using words as tokens,
0 => I’m1 => a2 => big3 => fat4 => developer
Using non-whitespace characters as tokens,
0 => I1 => ‘2 => m3 => a4 => b…16 => p17 => e18 => r
We could also consider all characters as tokens, to get
0 => I1 => ‘2 => m3 => (space)4 => a5 => (space)6 => b…20 => p21 => e22 => r
You get the idea, right?
Tokenizers (also called lexers) are used in the development of compilers for programming languages. They help the compiler make structural sense out of what you are trying to say. In this case, though, we’re building one for math expressions.
Tokens
A valid math expression consists of mathematically valid tokens, which for the purposes of this project could be Literals, Variables, Operators, Functions or Function Argument Separators.
A few notes on the above:
A Literal is a fancy name for a number (in this case). We’ll allow numbers in whole or decimal form only.
A Variable is the kind you’re used to in math: a,b,c,x,y,z. For this project, all variables are restricted to one-letter names (so nothing like var1 or price). This is so we can tokenize an expression like ma as the product of the variables m and a, and not one single variable ma.
Operators act on Literals and Variables and the results of functions. We’ll permit operators +, -, *, /, and ^.
Functions are “more advanced” operations. They include things like sin(), cos(), tan(), min(), max() etc
A Function Argument Separator is just a fancy name for a comma, used in a context like this: max(4, 5) (the maximum one of the two values). We call it a Function Argument Separator because it, well, separates function arguments (for functions that take two or more arguments, such as max and min).
Explanation:
Hope it help :D