About the article Nested Loops in Python All discussions about it

How to avoid slow nested loops?

TomBeckerOP

The article's star map example with two loops is clear, but my real data has 10,000 × 10,000 items and it's crawling.

6

Comments 5

Log in to join the discussion.

Log in Create account
  1. FreyaLreplies: 3

    100 million iterations in pure Python will be slow no matter what. What are you actually doing inside the loops?

    2
    1. TomBeckerOPreplies: 2

      For each sensor ID in list A, I check if it's in list B.

      0
      1. FreyaLreplies: 1

        Then you don't need nested loops at all. Convert B to a set once and check membership:

        b_ids = set(list_b)
common = [x for x in list_a if x in b_ids]

        That's 10,000 fast lookups instead of 100 million comparisons.

        1
        1. TomBeckerOP

          From several minutes to under a second. Wow.

          1
    2. HenryWrites

      Classic. Nested loops for "find matches" are almost always replaceable by a set or dict.

      4