Issues with free space managament of sparse files

[lvca]

During the test of a MT benchmark, the error Not enough disk space, force sync will be called popped up after a while. It looks like the WAL is never trunked. I have 53GB free on my SSD.

OrientDB v3.0.1 (develop branch).

Test case:

public class PerformanceIndexMTTest {
  public final static  String    DATABASE_PATH = "target/database";
  private static final int       TOT           = 5000000;
  private static final String    TYPE_NAME     = "Person";
  private static final ODocument END           = new ODocument();

  public static void main(String[] args) throws Exception {
    new PerformanceIndexMTTest().run();
  }

  private void run() {
    final File dir = new File(DATABASE_PATH + "/performance");
    OFileUtils.deleteRecursively(dir);
    dir.mkdirs();

    final int parallel = 2;

    OrientDB orient = new OrientDB("plocal:" + DATABASE_PATH, OrientDBConfig.defaultConfig());
    orient.create("performance", ODatabaseType.PLOCAL);
    final ODatabaseSession database = orient.open("performance", "admin", "admin");

    if (!database.getMetadata().getSchema().existsClass(TYPE_NAME)) {
      final OClass personType = database.getMetadata().getSchema().createClass(TYPE_NAME, parallel);
      personType.createProperty("id", OType.LONG);
      personType.createProperty("name", OType.STRING);
      personType.createProperty("surname", OType.STRING);
      personType.createProperty("locali", OType.INTEGER);
      personType.createProperty("notes1", OType.STRING);
      personType.createProperty("notes2", OType.STRING);

      personType.createIndex(TYPE_NAME + ".id", OClass.INDEX_TYPE.UNIQUE, "id");
    }

    long begin = System.currentTimeMillis();

    try {

      final Thread[] threads = new Thread[parallel];
      final ArrayBlockingQueue<ODocument>[] queues = new ArrayBlockingQueue[parallel];
      final AtomicInteger[] counter = new AtomicInteger[parallel];

      for (int i = 0; i < parallel; ++i) {
        queues[i] = new ArrayBlockingQueue<ODocument>(1024);
        counter[i] = new AtomicInteger();

        final int id = i;
        threads[i] = new Thread(new Runnable() {
          @Override
          public void run() {
            final ODatabaseSession database = orient.open("performance", "admin", "admin");
            while (true) {
              try {
                ODocument record = queues[id].take();

                if (record == END)
                  break;

                database.save(record, TYPE_NAME + (id == 0 ? "" : "_" + id));

                counter[id].incrementAndGet();

              } catch (InterruptedException e) {
                e.printStackTrace();
              }
            }

            OLogManager.instance().info(this, "Closing thread %d", id);
            database.close();
            OLogManager.instance().info(this, "Closed thread %d", id);
          }
        });
        threads[i].start();
      }

      long row = 0;
      for (; row < TOT; ++row) {
        final ODocument record = database.newInstance(TYPE_NAME);

        record.setProperty("id", row);
        record.setProperty("name", "Luca" + row);
        record.setProperty("surname", "Skywalker" + row);
        record.setProperty("locali", 10);
        record.setProperty("notes1",
            "This is a long field to check how OrientDB behaves with large fields. This is a long field to check how OrientDB behaves with large fields. This is a long field to check how OrientDB behaves with large fields. This is a long field to check how OrientDB behaves with large fields.");
        record.setProperty("notes2",
            "This is a long field to check how OrientDB behaves with large fields. This is a long field to check how OrientDB behaves with large fields. This is a long field to check how OrientDB behaves with large fields. This is a long field to check how OrientDB behaves with large fields.");

        try {
          queues[(int) (row % parallel)].put(record);
        } catch (InterruptedException e) {
          e.printStackTrace();
        }

        if (row % 100000 == 0)
          System.out.println("Written " + row + " elements in " + (System.currentTimeMillis() - begin) + "ms");

      }

      for (int i = 0; i < parallel; ++i) {
        try {
          queues[i].put(END);
        } catch (InterruptedException e) {
          e.printStackTrace();
        }
      }

      for (int i = 0; i < parallel; ++i) {
        try {
          threads[i].join();
        } catch (InterruptedException e) {
          e.printStackTrace();
        }
      }

      System.out.println("Inserted " + row + " elements in " + (System.currentTimeMillis() - begin) + "ms");

      System.out.println(" Counters ");
      for (int i = 0; i < parallel; ++i)
        System.out.println(" - " + counter[i].get());

    } finally {
      database.close();
      System.out.println("Insertion finished in " + (System.currentTimeMillis() - begin) + "ms");
    }
  }
}

Output:

Written 0 elements in 2ms
Written 100000 elements in 5960ms
Written 200000 elements in 9619ms
Written 300000 elements in 13380ms
Written 400000 elements in 17144ms
Written 500000 elements in 20878ms
Written 600000 elements in 25377ms
Written 700000 elements in 29063ms
Written 800000 elements in 33818ms
Written 900000 elements in 38141ms
Written 1000000 elements in 42160ms
Written 1100000 elements in 46283ms
Written 1200000 elements in 49956ms
Written 1300000 elements in 53719ms
Written 1400000 elements in 57607ms
Written 1500000 elements in 61355ms
Written 1600000 elements in 65282ms
Written 1700000 elements in 69089ms
Written 1800000 elements in 72991ms
Written 1900000 elements in 76780ms
Written 2000000 elements in 80554ms
Written 2100000 elements in 84349ms
Written 2200000 elements in 88107ms
Written 2300000 elements in 91972ms
Written 2400000 elements in 96154ms
Written 2500000 elements in 100732ms
Written 2600000 elements in 105795ms
Written 2700000 elements in 109984ms
Written 2800000 elements in 113812ms
Written 2900000 elements in 118791ms
Written 3000000 elements in 122881ms
Written 3100000 elements in 131169ms
Written 3200000 elements in 137366ms
Written 3300000 elements in 142464ms
Written 3400000 elements in 147055ms
Written 3500000 elements in 151473ms
Written 3600000 elements in 156005ms
Written 3700000 elements in 162993ms
Written 3800000 elements in 168244ms
Written 3900000 elements in 174288ms
Written 4000000 elements in 180323ms
Written 4100000 elements in 187034ms
Written 4200000 elements in 193305ms
Written 4300000 elements in 205683ms
Written 4400000 elements in 223523ms
Written 4500000 elements in 231899ms
Written 4600000 elements in 240256ms
Written 4700000 elements in 244627ms
Written 4800000 elements in 249298ms
May 03, 2018 1:00:32 PM com.orientechnologies.common.log.OLogManager log
SEVERE: $ANSI{green {db=performance}} Not enough disk space, force sync will be called
May 03, 2018 1:00:38 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Error during data write for file 'Person.id_66.sbt' 1-th attempt
java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

May 03, 2018 1:00:39 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Error during data write for file 'Person.id_66.sbt' 1-th attempt
java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

May 03, 2018 1:00:39 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Error during data write for file 'Person.id_66.sbt' 2-th attempt
java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

May 03, 2018 1:00:39 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Exception in thread 'OrientDB Write Cache Flush Task (performance)'
java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

May 03, 2018 1:00:39 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Exception `044A0D45` in storage `plocal:/Users/luca/work/dev/orientechnologies/orientdb/modules/orientdb-benchmarks/target/database/performance/performance`: 3.0.0 - Veloce (build 099033f72305d1c17f27d025b04afad7059c1298, branch develop)
com.orientechnologies.orient.core.exception.OWriteCacheException: File flush was abnormally terminated
	DB name="performance"
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1245)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1255)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeFullCheckpoint(OAbstractPaginatedStorage.java:3987)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.synch(OAbstractPaginatedStorage.java:3231)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceRequestsAndReadOnlyConditions(OAbstractPaginatedStorage.java:5349)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1911)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1873)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.internalCommit(ODatabaseDocumentAbstract.java:2721)
	at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.doCommit(OMicroTransaction.java:363)
	at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.commit(OMicroTransaction.java:166)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.endMicroTransaction(ODatabaseDocumentEmbedded.java:757)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.executeSaveRecord(ODatabaseDocumentEmbedded.java:747)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:244)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:174)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.saveInternal(ODatabaseDocumentAbstract.java:2059)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:2019)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:84)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2108)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2099)
	at com.orientechnologies.benchmarks.PerformanceIndexTest.run(PerformanceIndexTest.java:62)
	at com.orientechnologies.benchmarks.PerformanceIndexTest.main(PerformanceIndexTest.java:20)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1240)
	... 20 more
Caused by: java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

May 03, 2018 1:00:39 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Exception `044A0D45` in storage `plocal:/Users/luca/work/dev/orientechnologies/orientdb/modules/orientdb-benchmarks/target/database/performance/performance`: 3.0.0 - Veloce (build 099033f72305d1c17f27d025b04afad7059c1298, branch develop)
com.orientechnologies.orient.core.exception.OWriteCacheException: File flush was abnormally terminated
	DB name="performance"
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1245)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1255)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeFullCheckpoint(OAbstractPaginatedStorage.java:3987)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.synch(OAbstractPaginatedStorage.java:3231)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceRequestsAndReadOnlyConditions(OAbstractPaginatedStorage.java:5349)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1911)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1873)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.internalCommit(ODatabaseDocumentAbstract.java:2721)
	at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.doCommit(OMicroTransaction.java:363)
	at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.commit(OMicroTransaction.java:166)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.endMicroTransaction(ODatabaseDocumentEmbedded.java:757)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.executeSaveRecord(ODatabaseDocumentEmbedded.java:747)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:244)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:174)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.saveInternal(ODatabaseDocumentAbstract.java:2059)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:2019)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:84)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2108)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2099)
	at com.orientechnologies.benchmarks.PerformanceIndexTest.run(PerformanceIndexTest.java:62)
	at com.orientechnologies.benchmarks.PerformanceIndexTest.main(PerformanceIndexTest.java:20)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1240)
	... 20 more
Caused by: java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Insertion finished in 256764ms
Exception in thread "main" com.orientechnologies.orient.core.exception.ODatabaseException: Error during saving of record with rid #26:-2
	DB name="performance"
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:206)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.saveInternal(ODatabaseDocumentAbstract.java:2059)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:2019)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:84)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2108)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:2099)
	at com.orientechnologies.benchmarks.PerformanceIndexTest.run(PerformanceIndexTest.java:62)
	at com.orientechnologies.benchmarks.PerformanceIndexTest.main(PerformanceIndexTest.java:20)
Caused by: com.orientechnologies.orient.core.exception.OWriteCacheException: File flush was abnormally terminated
	DB name="performance"
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1245)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1255)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeFullCheckpoint(OAbstractPaginatedStorage.java:3987)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.synch(OAbstractPaginatedStorage.java:3231)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceRequestsAndReadOnlyConditions(OAbstractPaginatedStorage.java:5349)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1911)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1873)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.internalCommit(ODatabaseDocumentAbstract.java:2721)
	at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.doCommit(OMicroTransaction.java:363)
	at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.commit(OMicroTransaction.java:166)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.endMicroTransaction(ODatabaseDocumentEmbedded.java:757)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.executeSaveRecord(ODatabaseDocumentEmbedded.java:747)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:244)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:174)
	... 7 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flush(OWOWCache.java:1240)
	... 20 more
Caused by: java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
	at com.orientechnologies.common.io.OIOUtils.writeByteBuffer(OIOUtils.java:394)
	at com.orientechnologies.orient.core.storage.fs.OFileClassic.write(OFileClassic.java:226)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.flushPage(OWOWCache.java:2272)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.access$3500(OWOWCache.java:107)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2972)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache$FileFlushTask.call(OWOWCache.java:2939)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Process finished with exit code 1

[andrii0lomakin]

@lvca fixed. As I expected this issue is not related to WAL, but caused by the fact how *nix systems treat sparse files. So I have provided a simple fix (as usual hacks did not work and have to be removed :-) ) . If benchmarks will show speed degradation, I doubt that but still, I will provide a faster fix. Also, I have changed the title of the bug accordingly. Please check that issue is fixed.