Why is my legacy C# string parsing code much slower in .NET 8 compared to .NET Framework 4.5?

Your performance difference seems to have arisen from a change made in .NET 5, specifically a switch to the International Components for Unicode (ICU) localization library. From .NET globalization and ICU:

.NET globalization and ICU

Before .NET 5, the .NET globalization APIs used different underlying libraries on different platforms. On Unix, the APIs used International Components for Unicode (ICU), and on Windows, they used National Language Support (NLS). This resulted in some behavioral differences in a handful of globalization APIs when running applications on different platforms...

ICU on Windows

Windows now incorporates a preinstalled icu.dll version as part of its features that's automatically employed for globalization tasks. This modification allows .NET to use this ICU library for its globalization support.

If you switch back to NLS as shown in Use NLS instead of ICU,e.g. by editing your project file as follows, you should restore the previous performance.

XMLCopied
<ItemGroup>
  <RuntimeHostConfigurationOption Include="System.Globalization.UseNls" Value="true" />
</ItemGroup>

That being said, in comments you stated that GetTokens(string text) should not be localized. If that is the case, you need to modify your code as follows, as some of the string methods you are using are localized:

1. Pass StringComparison.Ordinal or StringComparison.OrdinalIgnoreCase to all calls to string.IndexOf(string, StringComparison).

2. Pass StringComparison.Ordinal or StringComparison.OrdinalIgnoreCase into string.StartsWith(string, StringComparison).

3. Replace string.ToUpper() with string.ToUpperInvariant().

4. Pass StringComparer.Ordinal or StringComparer.OrdinalIgnoreCase into all Enumerable.Contains<string>(, StringComparer) calls, e.g.:

   CSHARPCopied
   var comparer = _caseSensitive ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase;
   
   if (_keyWords.Contains(tk, comparer))
       token.TokenType = ZQLTokenType.KeyWord;
   

If you do this, switching back to NLS will not be required, and performance overall seems much faster. Demo fiddle here.

Finally, there are many ways this code could be optimized, for instance:

• Using a HashSet<string> for your keywords and operators.

• For large strings, you may end up allocating huge amounts of temporary memory when you do:

  rText = rText.Substring(idx + ftk.Length);

  You could replace this with a Span<char> or Memory<char> and eliminate this substring entirely.

However, this analysis is outside the scope of the question.

Full code here:

CSHARPCopied
public class myclass
{
    int _lineCounter = 0;
    int _indexCounter = 0;
    char[] _separators = new char[] { ' ', '.', ',', '\n', '\t', '\r', '(', ')' };
    static string[] _operators = new string[] { "!=", "=", "<>", "<", ">", " + " };
    static string[] _lOperators = new string[] { "!=", "=", "<>", "<", ">", " + " };

    bool _caseSensitive = false;
    List<string> _keyWords = new List<string>(new string[] { "NOT_EMPTY", "DISTINCT", "TABLE", "DROP", "APPEND", "SELECT", "WHERE", "FROM", "AND", "OR", "AS", "INTO", "XML_ATT", "XML_VAL", "GROUP", "BY", "ORDER", "NUMBER" });

    public List<ZQLToken> GetTokens(string text)
    {
        List<ZQLToken> ret = new List<ZQLToken>();
        string rText = text;

        _indexCounter = 0;
        _lineCounter = 1;

        string[] tks = text.Split(_separators, StringSplitOptions.RemoveEmptyEntries);

        foreach (string ftk in tks)
        {
            char pSep = '\0';
            char nSep = '\0';

            int idx = rText.IndexOf(ftk, StringComparison.Ordinal);
            if (idx > 0)
                pSep = rText[idx - 1];
            if (idx + ftk.Length < rText.Length)
                nSep = rText[idx + ftk.Length];

            rText = rText.Substring(idx + ftk.Length);
            _indexCounter = text.IndexOf(rText, StringComparison.Ordinal);

            string[] moreTk = ftk.Split(_operators, StringSplitOptions.RemoveEmptyEntries);

            string startOp = "";
            foreach (string oper in _operators)
            {
                if (ftk.StartsWith(oper, StringComparison.Ordinal))
                    startOp = oper;
            }

            if (startOp != "")
            {
                string tok = ftk.Substring(startOp.Length);
                AddToken(ret, startOp, pSep, '\0');
                AddToken(ret, tok, '\0', nSep);
            }
            else if (moreTk.Length >= 2)
            {
                string rFtk = ftk;
                string tk1 = moreTk[0];
                AddToken(ret, tk1, pSep, '\0');
                for (int i = 1; i < moreTk.Length; i++)
                {
                    string tk2 = moreTk[i];
                    int idx2 = rFtk.IndexOf(tk2, tk1.Length, StringComparison.Ordinal);

                    string op = rFtk.Substring(tk1.Length, Math.Abs(idx2 - tk1.Length));

                    AddToken(ret, op, '\0', '\0');

                    if (i + 1 == moreTk.Length)
                        AddToken(ret, tk2, '\0', nSep);
                    else
                        AddToken(ret, tk2, '\0', '\0');

                    rFtk = rFtk.Substring(idx2);
                    tk1 = tk2;
                }
            }
            else
                AddToken(ret, moreTk[0], pSep, nSep);
        }

        return ret;
    }

    private void AddToken(List<ZQLToken> ret, string tk, char pSep, char nSep)
    {
        if (tk == "")
            return;

        ZQLToken token = new ZQLToken();
        token.Line = _lineCounter;
        token.Index = _indexCounter;
        if (ret.Count > 0)
        {
            ZQLToken ptok = ret[ret.Count - 1];
            ptok.NextToken = token;
            token.PrevToken = ptok;
        }

        token.PrevSeparator = pSep;
        token.NextSeparator = nSep;

        var comparer = _caseSensitive ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase;
        
        if (_keyWords.Contains(tk, comparer))
            token.TokenType = ZQLTokenType.KeyWord;
        else if (_lOperators.Contains(tk, comparer))
            token.TokenType = ZQLTokenType.Operator;
        else
            token.TokenType = ZQLTokenType.Variable;

        token.Text = tk;

        if (token.NextSeparator == '\n')
            _lineCounter++;

        ret.Add(token);
    }
}

---

Update OK, I made some benchmarks using BenchmarkDotNet for the following test class:

CSHARPCopied
public class TestClass
{
    [Benchmark]
    public void Test_myClass_GetTokens()
    {
        var text = GetInput(); // The input string from the question
        
        var myClass = new myclass();
        var list = myClass.GetTokens(text);

        Assert.That(list.Count == 4821);
    }

Here are the results on .NET 8, x64, Release build:

1. Original code, ICU:

   | Method                 | Mean    | Error    | StdDev   |
   |----------------------- |--------:|---------:|---------:|
   | Test_myClass_GetTokens | 3.778 s | 0.0420 s | 0.0372 s |
   
   

2. Original code, NLS:

   | Method                 | Mean     | Error   | StdDev  |
   |----------------------- |---------:|--------:|--------:|
   | Test_myClass_GetTokens | 246.9 ms | 2.78 ms | 2.60 ms |
   
   

3. Fixed Ordinal code from my answer, ICU:

   | Method                 | Mean     | Error    | StdDev   |
   |----------------------- |---------:|---------:|---------:|
   | Test_myClass_GetTokens | 17.62 ms | 0.218 ms | 0.182 ms |
   
   

As you can see, ICU is roughly 15 times slower than NLS for the original code in the question -- but switching to ordinal (using ICU) is 14 times faster than the original, localized code using NLS. And since the method isn't supposed to be culture-sensitive anyway, using Ordinal or OrdinalIgnoreCase everywhere would seem to be the correct fix in both .NET Core and .NET Framework.