<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Whack First]]></title><description><![CDATA[Static, a whack, then a clear picture. Sharing what we learn about Python, Excel and everyday tech.]]></description><link>https://whackfirst.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Fri, 20 Nov 2026 15:28:34 GMT</lastBuildDate><atom:link href="https://whackfirst.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[pandas read_excel treats the title as the header: 'Unnamed: 1' columns and a total that is too low]]></title><description><![CDATA[The script printed Done with a total of 12,500 instead of 15,000. Then the repair crashed with a to_excel TypeError.]]></description><link>https://whackfirst.hashnode.dev/pandas-read-excel-treats-the-title-as-the-header-unnamed-1-columns-and-a-total-that-is-too-low</link><guid isPermaLink="true">https://whackfirst.hashnode.dev/pandas-read-excel-treats-the-title-as-the-header-unnamed-1-columns-and-a-total-that-is-too-low</guid><category><![CDATA[pandas]]></category><category><![CDATA[Python]]></category><category><![CDATA[excel]]></category><category><![CDATA[openpyxl]]></category><category><![CDATA[debugging]]></category><dc:creator><![CDATA[whackfirst]]></dc:creator><pubDate>Sun, 08 Nov 2026 14:17:18 GMT</pubDate><content:encoded><![CDATA[<p>If a merged Excel total is lower than it should be and the combined DataFrame has columns named <code>Unnamed: 1</code>, <code>Unnamed: 2</code> and so on, one of your workbooks probably has a report title above the real header row. <code>read_excel</code> uses the first row as the header by default, so the title becomes a column name and the real headers become data. Nothing raises an error. In this run the script printed Done and a total of 12,500 where 15,000 was correct.</p>
<p>Test conditions: Windows 11, Python 3.14.7, pandas 3.1.0, openpyxl 3.1.5. The workbooks are small synthetic fixtures, not customer data. Exact error wording can differ with versions.</p>
<h2>The symptom</h2>
<p>The South workbook had a report title in the first row, a blank row, and then the real headers. The North workbook was normal. The merge script printed:</p>
<pre><code class="language-text">📄 Read north.xlsx (1 rows)
📄 Read south.xlsx (3 rows)
✅ Done! Saved 4 rows to 'merged.xlsx'.
💰 Total sales: 12,500.0
</code></pre>
<p>The input had two orders, so 4 rows is already wrong, and South reading as 3 rows for one order is a hint. A separate check, outside the merge program, gave the expected and observed values:</p>
<pre><code class="language-text">Independent expected rows: 2; expected amount: 15000
Observed rows: 4; observed amount: 12500
Columns: ['OrderDate', 'OrderID', 'Product', 'Quantity', 'Amount', 'SourceFile', 'South branch monthly sales', 'Unnamed: 1', 'Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4']
</code></pre>
<p>The column <code>South branch monthly sales</code> is the title text, used as a header. The <code>Unnamed</code> columns are the empty cells to its right. An <code>Unnamed</code> column is a clue, not a diagnosis, so confirm it against the original workbook.</p>
<h2>Look at the raw top rows</h2>
<p>Read the file without letting pandas pick a header:</p>
<pre><code class="language-python">import sys
import pandas as pd

df = pd.read_excel(sys.argv[1], header=None, nrows=6)
print(df.to_string())
</code></pre>
<p>On South's workbook:</p>
<pre><code class="language-text">                            0        1        2         3       4
0  South branch monthly sales      NaN      NaN       NaN     NaN
1                         NaN      NaN      NaN       NaN     NaN
2                   OrderDate  OrderID  Product  Quantity  Amount
3         2026-10-04 00:00:00   S-1001   Coffee         1    2500
</code></pre>
<p>The real header is on the third row. With the title treated as the header, the real <code>Amount</code> values sit in the column that pandas named <code>Unnamed: 4</code>, which is why the <code>Amount</code> column has nothing for South.</p>
<p>To see the effect per file, this script groups the merged workbook by <code>SourceFile</code>:</p>
<pre><code class="language-python">import sys
import pandas as pd

df = pd.read_excel(sys.argv[1])
print("Columns:", list(df.columns))
amount = pd.to_numeric(df["Amount"], errors="coerce")
table = pd.DataFrame({
    "Rows": df.groupby("SourceFile").size(),
    "Amount total": amount.groupby(df["SourceFile"]).sum(),
    "Missing amount": amount.isna().groupby(df["SourceFile"]).sum(),
})
print(table)
print("Total rows:", len(df), "/ Total amount:", f"{amount.sum():,.0f}")
</code></pre>
<pre><code class="language-text">Columns: ['OrderDate', 'OrderID', 'Product', 'Quantity', 'Amount', 'SourceFile', 'South branch monthly sales', 'Unnamed: 1', 'Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4']
            Rows  Amount total  Missing amount
SourceFile                                    
north.xlsx     1       12500.0               0
south.xlsx     3           0.0               3
Total rows: 4 / Total amount: 12,500
</code></pre>
<p>South contributes 3 rows and no amount. A per-file comparison like this shows where the money went, which a grand total cannot.</p>
<h2>The direct fix</h2>
<p><code>header</code> counts from zero, and the raw output above shows the real header at index 2. For this file, that is the whole fix:</p>
<pre><code class="language-python">df = pd.read_excel("south.xlsx", header=2)
</code></pre>
<p>On South's workbook this gave the five expected columns, 1 row and an amount of 2500. A fixed number only works while every file has the same layout. If the title rows differ from file to file, find the header by its column names, which is what the repair below tried.</p>
<h2>The repair attempt</h2>
<p>The repair prompt gave the model the expected and observed numbers, the raw top rows, the code and the output, and asked for the header to be found by required column names and for rejected rows to be reported. The returned code found the header with this function and its helpers:</p>
<pre><code class="language-python">REQUIRED = ["OrderDate", "OrderID", "Product", "Quantity", "Amount"]
REQ_KEY = {c.casefold(): c for c in REQUIRED}

def norm(v):
    return "" if v is None else str(v).strip().casefold()

def find_header(rows):
    """Return (index, {canonical name: column index}) of the first row holding all required names."""
    for i, row in enumerate(rows):
        pos = {}
        for j, v in enumerate(row):
            k = norm(v)
            if k in REQ_KEY and REQ_KEY[k] not in pos:
                pos[REQ_KEY[k]] = j
        if len(pos) == len(REQUIRED):
            return i, pos
    return None, None
</code></pre>
<p><code>rows</code> is the sheet as a list of tuples, from openpyxl's <code>ws.iter_rows(values_only=True)</code>. In a separate check on South's workbook, this function returned index 2, and passing that index to <code>read_excel</code> as <code>header</code> gave 1 row and an amount of 2500.</p>
<p>The first lines of its run were correct. The program read South properly: 1 order and 2 rejected rows. The side run below names those rows as the title and the blank row.</p>
<pre><code class="language-text">Read north.xlsx[Sheet]: 1 orders, 0 rejected rows
Read south.xlsx[Sheet]: 1 orders, 2 rejected rows
</code></pre>
<p>Then it crashed at the last step, writing the workbook (traceback trimmed):</p>
<pre><code class="language-text">    od.to_excel(xw, "Orders", index=False)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: NDFrame.to_excel() takes 2 positional arguments but 3 positional arguments (and 1 keyword-only argument) were given

During handling of the above exception, another exception occurred:

    raise IndexError("At least one sheet must be visible")
IndexError: At least one sheet must be visible
</code></pre>
<p>The failing line passes the sheet name by position: <code>od.to_excel(xw, "Orders", index=False)</code>. The pandas 3.1.0 installed here rejected it. The question did not state the pandas version, and whether stating it would have changed the answer was not tested. This article also does not say which release introduced the rule. The second exception follows because the crash happened inside <code>with pd.ExcelWriter(...)</code>. The traceback shows the writer's close step calling openpyxl's save, which raised because no sheet was visible. A <code>merged.xlsx</code> was left behind, and openpyxl could not open it.</p>
<h2>The result</h2>
<p>The checks were fixed before the model was asked, and this answer failed all three: the workbook with the title row, normal workbooks, and a workbook that also had a repeated header and a subtotal row. The crash was the same on normal workbooks, because it sits in the save step, not in the header logic. The answer looked complete, but with no tools the model could not have run it.</p>
<p>A side run, not scored, changed only the three <code>to_excel</code> calls from the positional form to <code>sheet_name=</code>:</p>
<pre><code class="language-python"># Before
od.to_excel(xw, "Orders", index=False)
# After
od.to_excel(xw, sheet_name="Orders", index=False)
</code></pre>
<p>For the bad fixture it printed:</p>
<pre><code class="language-text">Read north.xlsx[Sheet]: 1 orders, 0 rejected rows
Read south.xlsx[Sheet]: 1 orders, 2 rejected rows

SourceFile  Orders  AmountTotal
north.xlsx       1      12500.0
south.xlsx       1       2500.0

Total: 2 orders, 15,000
Rejected rows: 2 (see 'Rejected' sheet)
Reason
title/notes before header    1
blank row before header      1
Saved to merged.xlsx
</code></pre>
<p>That is the expected 2 orders and 15,000. It is a patched variant, not what the model returned, and the scored answer stays the unedited one. The code came from one model, Claude Sonnet 5.5, in one scored run with no tools, so it says nothing about other models or your files. An earlier run under the wrong language setting was discarded before any check was run.</p>
<h2>Before you accept a repair</h2>
<p>Run the repaired program against the original bad workbook and against a normal workbook that worked before, and compare rows and totals per file with a count made independently of the program. If the save step throws, delete any <code>merged.xlsx</code> it left, because a half-written file can be mistaken for output.</p>
<p>This case is one of ten in <a href="https://whackfirst.gumroad.com/l/when-ai-written-python-fails">When AI-Written Python Fails</a>, a book of reproduced spreadsheet and web-data failures with repair prompts.</p>
]]></content:encoded></item><item><title><![CDATA[TypeError: '<' not supported between instances of 'str' and 'Timestamp' when merging Excel files in pandas]]></title><description><![CDATA[One workbook holds a real date, another holds the text 2026.10.04. How to see it, and a repair that refuses to guess.]]></description><link>https://whackfirst.hashnode.dev/typeerror-not-supported-between-instances-of-str-and-timestamp-when-merging-excel-files-in-pandas</link><guid isPermaLink="true">https://whackfirst.hashnode.dev/typeerror-not-supported-between-instances-of-str-and-timestamp-when-merging-excel-files-in-pandas</guid><category><![CDATA[pandas]]></category><category><![CDATA[Python]]></category><category><![CDATA[excel]]></category><category><![CDATA[openpyxl]]></category><category><![CDATA[debugging]]></category><dc:creator><![CDATA[whackfirst]]></dc:creator><pubDate>Sun, 08 Nov 2026 14:15:51 GMT</pubDate><content:encoded><![CDATA[<p>If <code>sort_values</code> fails with <code>TypeError: '&lt;' not supported between instances of 'str' and 'Timestamp'</code> after you combined several Excel files, the column you are sorting holds both real dates and text. In this run, one file stored its dates as text while another stored real Excel dates, so the combined column held both <code>Timestamp</code> objects and <code>str</code> objects, and Python cannot order those against each other. The repair shown here looks at the types per file, then converts only the text formats it can name.</p>
<p>Test conditions: Windows 11, Python 3.14.7, pandas 3.1.0, openpyxl 3.1.5. The workbooks are small synthetic fixtures, not customer data. Exact error wording can differ with versions, so quote the message you actually see.</p>
<h2>The script that fails</h2>
<p>This folder-merge script is deliberately fragile. It reads every <code>.xlsx</code>, tags each row with its file, concatenates, sorts by date and saves.</p>
<pre><code class="language-python">for file_name in os.listdir(folder_path):
    if file_name.endswith(".xlsx"):
        file_path = os.path.join(folder_path, file_name)
        df = pd.read_excel(file_path)
        df["SourceFile"] = file_name
        all_data.append(df)
        print(f"📄 Read {file_name} ({len(df)} rows)")
merged_df = pd.concat(all_data, ignore_index=True)
merged_df = merged_df.sort_values(by="OrderDate")
merged_df.to_excel(output_file, index=False)
print(f"✅ Done! Saved {len(merged_df)} rows to '{output_file}'.")
print(f"💰 Total sales: {merged_df['Amount'].sum():,}")
</code></pre>
<p>In this run the North workbook had a real Excel date in <code>OrderDate</code>. The South workbook had the string <code>2026.10.04</code> in the same column. Both files read without complaint, and the failure came at the sort (traceback trimmed):</p>
<pre><code class="language-text">📄 Read north.xlsx (1 rows)
📄 Read south.xlsx (1 rows)
    merged_df = merged_df.sort_values(by="OrderDate")
TypeError: '&lt;' not supported between instances of 'str' and 'Timestamp'
</code></pre>
<p>Nothing in the read step warned about this. In this run <code>read_excel</code> kept the South value as <code>str</code>, and the combined column kept both types.</p>
<h2>A second failure that saves first</h2>
<p>Amounts can have the same problem. In a separate run both dates were real, so the sort passed, but the South amount was the text <code>$2,500</code>. The program printed its success line and then failed (traceback trimmed):</p>
<pre><code class="language-text">📄 Read north.xlsx (1 rows)
📄 Read south.xlsx (1 rows)
✅ Done! Saved 2 rows to 'merged.xlsx'.
TypeError: unsupported operand type(s) for +: 'int' and 'str'
</code></pre>
<p>Note the order. <code>merged.xlsx</code> was already written when the total failed, so a file exists on disk that came from a run that ended in an error. If something downstream picks up that file, it picks up a workbook whose <code>Amount</code> column contains text. Checking that the run exited cleanly matters as much as checking that the file exists.</p>
<h2>See the types before changing anything</h2>
<p>Excel's display format does not tell you what is stored. A cell that shows <code>2026-10-04</code> can be a date or a string. This short script counts the Python type of every non-empty value per column:</p>
<pre><code class="language-python">import sys
from collections import Counter
import pandas as pd

for filename in sys.argv[1:]:
    df = pd.read_excel(filename)
    print(f"[{filename}]")
    for column in df.columns:
        counts = Counter(type(v).__name__ for v in df[column].dropna())
        print(f"  {column}: {dict(counts)}")
</code></pre>
<p>Run on the two date-case workbooks, the output was (folder paths removed):</p>
<pre><code class="language-text">[north.xlsx]
  OrderDate: {'Timestamp': 1}
  OrderID: {'str': 1}
  Product: {'str': 1}
  Quantity: {'int': 1}
  Amount: {'int': 1}
[south.xlsx]
  OrderDate: {'str': 1}
  OrderID: {'str': 1}
  Product: {'str': 1}
  Quantity: {'int': 1}
  Amount: {'int': 1}
</code></pre>
<p><code>Timestamp</code> in one file and <code>str</code> in the other, for the same column, is the whole diagnosis. On larger files the counts also show whether the text values are a few rows or most of the column.</p>
<h2>A repair that does not guess</h2>
<p>A blanket conversion decides for you what an unreadable value means. A day-first versus month-first ambiguity is the clearest case: <code>03/04/2026</code> is a valid date under either convention and means a different day under each.</p>
<p>The repair below came from a prompt that listed the exact formats present and said to normalize only those, not to guess ambiguous dates, and to record unreadable values in a review sheet. Two functions carry the design. These are excerpts, starting with the date parser.</p>
<pre><code class="language-python">DATE_TEXT = re.compile(r"^(\d{4})\.(\d{2})\.(\d{2})$")
AMOUNT_TEXT = re.compile(r"^\$(\d{1,3}(?:,\d{3})+|\d+)(\.\d{1,2})?$")

def parse_date(value):
    """Return (date, None) on success or (None, reason) on failure."""
    if _is_blank(value):
        return None, "missing value"
    if isinstance(value, (dt.date, dt.datetime, pd.Timestamp)):
        return pd.Timestamp(value).date(), None
    if isinstance(value, str):
        match = DATE_TEXT.match(value.strip())
        if not match:
            return None, "unsupported date format"
</code></pre>
<p>The text branch of the amount parser:</p>
<pre><code class="language-python">    if isinstance(value, str):
        text = value.strip()
        if not AMOUNT_TEXT.match(text):
            return None, "unsupported amount format"
        return Decimal(text.replace("$", "").replace(",", "")), None
</code></pre>
<p>Each function returns either a value or a reason. A string that matches <code>YYYY.MM.DD</code> and is a real calendar date becomes a date. A string matching <code>$</code> plus digits with optional thousands separators becomes a <code>Decimal</code>. Everything else, including <code>03/04/2026</code> and <code>abc</code>, comes back with a reason, and the row goes to a <code>NeedsReview</code> sheet with the source file, the original row number and the original value. The row is left out of the merged sheet, and the program prints how many rows were excluded, so omitted money is visible instead of silent.</p>
<h2>What happened when it ran</h2>
<p>With South's date as <code>2026.10.04</code> and its amount as <code>$2,500</code>, the repaired program printed:</p>
<pre><code class="language-text">📄 Read north.xlsx (1 rows)
📄 Read south.xlsx (1 rows)
✅ Done! Saved 2 rows to 'merged.xlsx'.
💰 Total sales (merged rows only): 15,000
⚠️ Excluded rows: 0
   Readable amounts inside excluded rows: 0
   Rows whose amount was unreadable (not in any total): 0
🔎 NeedsReview entries: 0
</code></pre>
<p>That is 2 rows, a total of 15,000, real dates and a sorted result. With <code>03/04/2026</code> as the South date, the value was not guessed; <code>NeedsReview</code> listed <code>south.xlsx</code>, row 2 and the value. With <code>abc</code> as the amount, it appeared in <code>NeedsReview</code> with file and row 2, and the program printed 1 excluded row and 1 unreadable amount.</p>
<p>Treat this as one answer, not a success rate. The code came from one model, Claude Sonnet 5.5, in one scored run with no tools. An earlier run under the wrong language setting was discarded before any check was run. One answer does not show that the code works on another model or on your files. A review sheet also does not replace checking the excluded rows yourself, because an excluded row is money that is missing from the total. Decide your date convention before you rely on any of this, and test with one deliberately bad date and one deliberately bad amount.</p>
<p>This case is one of ten in <a href="https://whackfirst.gumroad.com/l/when-ai-written-python-fails">When AI-Written Python Fails</a>, a book of reproduced spreadsheet and web-data failures with repair prompts.</p>
]]></content:encoded></item></channel></rss>