Multiple Codes in one image

[boruchsiper]

Would be nice if decodeFromVideoDevice/decodeFromImage could return an array result when multiple codes are detected.

Here is an example

[odahcam]

I know that's possible for PDF417, not sure about the rest. It is certainly something to look for since more people are asking for it.

[thorgch]

The company I work for is willing to pay to have this capability, and is also willing to keep that work open sourced in accordance with the terms of the license. Our application needs to scan multiple code128 and/or QRs in a single image or video stream. We’ve looked into commercial/enterprise libraries that can do it, but we figured why not just take that money and pay this team to get this feature in place. So, any suggestions by the contributors of this library on how we could get that going?

[jokester]

I wanted the capacity to read multiple QRs in one image too. // In case anyone wonders, for a small tool.

This is what I found by far:

• The QR related classes (QRCodeReader qrcode/Decoder and especially qrcode/Detector) do not assume multiple QR in 1 image. For an image of 2 vertically aligned QR, it would find 4 Finder Patterns (from 2 QR codes) in the left, and fail with a checksum error.
• It should be possible to group Find Patterns from their positions (I'm not a CV guru though), and feed each group to Decoder to extract QRs correctly.

One thing is, I'm not sure if such changes or hacks are welcome in this repository: supporting multiple QRs would require new APIs, and it started as a port of java ZXing after all.

[thorgch]

@jokester I see your point about wanting to preserve the original API, especially in the spirit of the original port. With that said, I wonder if it’s conceivable to have a new “extended” API that allows for this capability without affecting the original.

[odahcam]

@thorgch I agree with you and I think we could have a extended API. Also, I can't remember by heart by I'm pretty sure some decoder in here does support multiple barcode reading from the same image, so that feature wouldn't be that hard to implement I think.

Edit.: I commented right above whats the decoder, it's PDF417 decoder.

[thorgch]

@odahcam If you think it’s possible, then I’m serious about paying to bump this feature request to the head of the queue. We’re specifically after the ability the read multiple QR or Code128 codes. I see there are various platforms to sponsor this project, and I’d be happy to use any one of them if this project’s team has a preference. I’m also open to something more formal like a contract if this ends up being a non-trivial amount of work and you want a guarantee of payment for the work done. I also want to reiterate that we’d expect this work to remain opensource under an MIT license. If you, or one of the other contributors are interested, let me know and I’ll email directly to sort out the payment details.

[odahcam]

read multiple QR or Code128 codes

I'm just not sure if we can do this fast, so it's delicate to accept payments to develop this specific stuff even if it would help the project a lot by enabling me to spend more hours here, but it stills complicated. Sorry.

[CaueP]

Hello, folks! The project I'm currently working is also in need of a multiple barcode and QR Code reader, so I'm very interested in this issue too.

I saw that there are PRs in progress and I have a couple questions about what is the goal of this issue because I'm not very familiar with the project components to fully understand the PRs:

1. Will it be available for video devices (such as BrowserMultiFormatReader.decodeFromVideoDevice)?
2. Is this solution covering multiple 1D barcodes too (e.g. code 128)?

Besides that, is there any help needed to advance the implementation? I'm willing to help on coding too if it's within my reach.

[kevinhikaruevans]

If anybody is still watching this, I wrote a temporary workaround for vertically-stacked 1D barcodes. I'm basically copying the camera input into a canvas, iterating over like several (5ish) horizontal strips and then scanning each strip. It's pretty inefficient but it works for my needs.

I'll probably use a continuous video scanner then when a barcode is detected, then it should break up the frame and scan for other barcodes.

[kevinhikaruevans]

Here's what the component looks like. I had to clear out some work-related code, so it might not work at first. You'll probably have to adjust some things to get it working.

import React, { useState, useEffect, useRef } from 'react';
import { BrowserMultiFormatReader, BarcodeFormat } from '@zxing/browser';

// number of rescan blocks to split the video into
const BLOCKS = 5;

const SERIAL_REGEX = /.*/g; 

const DEFAULT_VIDEO_CONSTRAINTS = {
    audio: false,
    video: {
      facingMode: 'environment',
    //   width: 720,
    //   height: 640
    }
  };

const barcodeReader = new BrowserMultiFormatReader();

function isValidSerial(value : string) {
  return SERIAL_REGEX.test(value);
}


type Props = {
    scanning: boolean,
    onChange: (value: Object) => void
};

/**
 * A component that scans either a GM 2D barcode *or* a vertically stacked list of 1D barcodes.
 */
export default function MultiSerialScanner(props : Props) {
    const { scanning, onChange } = props;
    const videoRef = useRef(null);
    
    const [ serial, setSerial ] = useState('');

    if (!onChange) {
        throw new Error('onChange is not defined');
    }

    useEffect(() => {
        // detect changes in any of the scanned variables
        onChange({ serial });
    }, [serial, onChange]);

    useEffect(() => {
        if (scanning) {
            // reset anytime scanning is turned back on
            reset();
        } else {
            // barcodeReaderPromise.stop();
            // stop scanning if the parent sets scanning=false
            videoRef.current.srcObject.getTracks()[0].stop();
        }
    }, [scanning]);

    useEffect(() => {
        let video : MediaStream = null;

        if (navigator.mediaDevices === undefined) {
            console.error('Browser does not support mediaDevices. Scanner will not work. Are you using a secure environment?');
            return;
        }

        navigator.mediaDevices.getUserMedia(DEFAULT_VIDEO_CONSTRAINTS)
            .then(async (stream) => {
                barcodeReader.decodeFromStream(stream, videoRef.current, handleDecode);
                
                if (videoRef.current) {
                    videoRef.current.srcObject = video = stream;
                }
            })
            .catch(err => {
                console.error("failed to get camera feed", err);
            });

        return () => {
            // clean up
            video.getTracks()[0].stop();

        };
    });


    /**
     * Resets any scanned values back to empty strings.
     */
    function reset() {
        setSerial('');
    }

    /**
     * Handles when a single 1D barcode is decoded.
     * @param {string} text The barcode text
     */
    function onFoundPartialSerial(text : string) {
        if (isValidSerial(text)) {
            setSerial(text);
        }
    }

    /**
     * Scans a single frame for multiple barcodes by splitting the frame
     * into blocks and scanning each block individually.
     */
    function scanSingleFrame() {
        const canvas = document.createElement('canvas');
        const context = canvas.getContext('2d');
        const blockHeight = videoRef.current.videoHeight / BLOCKS;
        
        canvas.width = videoRef.current.videoWidth;
        canvas.height = blockHeight;

        for (let index = 0; index < BLOCKS; index++) {
            context.drawImage(
                videoRef.current,
                0, index * blockHeight,
                videoRef.current.videoWidth, blockHeight,
                0,
                0,
                canvas.width, canvas.height
            );
            const dataUrl = canvas.toDataURL('image/png');
            // imgRefs.current[index].src = dataUrl;
            barcodeReader.decodeFromImageUrl(dataUrl).then(result => {
                const { text } = result;
                console.log('found rescanned barcode', result);
                onFoundPartialSerial(text);
            }).catch(err => {
                // console.error(err);
            });
        }
    }

    /**
     * Handles decoding a single barcode
     * @param {*} result 
     */
    function handleDecode(result : object) {
        if(!result) {
            return;
        }
        const { text } = result;

        switch(result.format) {
            case BarcodeFormat.QR_CODE:
                

                break;

            case BarcodeFormat.CODE_128:
                console.log('found CODE_128 barcode: ' + result.text);
                onFoundPartialSerial(text);

                // scan for other barcodes in the frame
                scanSingleFrame();
                break;
            
            default:
                console.error('Unknown barcode format:', result.format);
        }        
    }

    return <>
        <video ref={videoRef} autoPlay playsInline muted style={{height: 400, width: 400}}/><br />
        <div>
            {serial}
        </div>
    </>;
}