Why an Example-Based Workflow?
Sustainability. It’s a word that comes up a lot throughout many facets of life. When rapid progress occurs in an industry, the truly disruptive kind, instabilities are created and unstable systems don’t last. We see many examples of this around the world, but rarely think about what it means and how it could be happening within the computer graphics industry.
Computer graphics is rapidly progressing, which in turn leads to an unstable supply and demand market. Our demand to consume 3D content has long outpaced our ability to produce it. We see many symptoms of this inequality: exponentially growing team sizes, ballooning project budgets and durations, along with dependence on distributed teams and Far East outsourcing. These are cracks in the industries foundation. The brute force solution can’t last.
Figure 1: The rising cost of game development is exponential.
Thus, we make a case for a new paradigm in content creation, a shift in mindset as well as technology stack: the Example-Based Workflow.
What is an Example-Based Workflow?
The philosophy behind an Example-Based workflow is that we can’t keep painting and programming every pixel. Rather than starting from a blank canvas and building everything from scratch using a manual or procedural workflow, start with an example of what is needed e.g. a scan, a photo or an older asset from a previous generation. The user should start from raw examples, giving the computer (1) data and (2) high-level instructions on what to do with that data, then (3) the computer automates some otherwise manual process.
Figure 2: The Example-Based workflow
To make things simple, here’s a list of the three ingredients that make up an Example-Based Workflow:
- Start from seed data or an example of what the artist wants
- Human provides the intent
- OPTIONAL: Refinement through human feedback, return to step 2
This might sound unconventional, but the industry is already using an Example-Based workflow today. Photogrammetry, the conversion of 2D photos into textured 3D models, although a niche use case, can be considered the first wave of Example-Based content creation, leveraging data to automate the 3D modeling process. We at Artomatix see Creative AI is the second wave, using ArtEngine as a case study, Example-Based content creation is moving downstream in the production pipeline, further automating the clean-up and modification of material data. Down the line even further we see the building of 3D worlds.
Case Study – Unity Fontainebleau demo: Unity has a big mission – the democratization of game development. This means that anyone can make a game, regardless of budget or background. Unity needs to think differently in all aspects of the development pipeline, including content creation. When Unity decided to explore Photogrammetry and how that would fit into their engine, they decided from the start that they would do so with a small team and a strict timeline, this necessitated adopting the most efficient methods possible – brute force wasn’t an option. Most studios see scanning as a means to create highly photorealistic content very quickly. Studios are investing in scanning equipment and associated software such as Reality Capture which has gained an excellent reputation. That said, Photogrammetry is not a fully turnkey solution as manual cleanup and tweaking is needed to go from raw scan to production ready asset. This is where Unity had to develop and explore other solutions, their own technical artist Cyril Jover developed the industry’s leading Delighting tool designed specifically for Photogrammetric scans. As part of this effort, the team adopted ArtEngine for automated seam removal and the general scan cleaning process.
Figure 3: Unity Fontainebleau Demo
To further this point, Microsoft technical artist Pete McNally is similarly charged with the task of building realistic environments with a small team. These constraints led Pete to become an industry pioneer in adopting and utilizing an Example-Based workflow. Pete shared his full thoughts on the topic at 80lvl:
“There are already some solutions for removing seams on materials but I haven’t found anything to match ArtEngine by Artomatix. It uses an example-based approach, rather than procedural to remove seams and can also mutate textures, creating new textures based on input from existing ones. For example, I used Artomatix in the Сraggy Cliffs:
Figure 4: Mutation and Seam Removal on Photogrammetry scans courtesy of Pete McNally
It has given a new lease of life to some 3D scans I’ve done that were too problematic to be used in production, such as textures that were too blurry or geometry that was stretched or had holes in it. ArtEngine can fill in missing detail, allowing you to paint areas that you’d like to ignore and producing new viable textures.”
So that’s what we mean by an Example-Based workflow, the computer extrapolates and ideates upon example data, so the artist provides the high-level sophisticated creativity and the computer contributes the low-level mechanistic creativity. It’s a perfect symbiosis between artist and tool, where they empower each other. This isn’t just a necessary evolution for the computer graphics industry, it is an inevitable one.
How Does it Work?
If you’re still reading, you’re probably wondering how you would use an Example-Based workflow in practice. Figure 2 identified six pillars to the workflow, three on the data side and three that automate. While each of these pillars warrants their own blog post, I’d like to briefly touch on automation by highlighting the three tasks that should no longer require manual intervention, as this is likely foreign to many artists.
The Industry is sitting on mountains of old, outdated 3D content. Every five years the standard resolution for textures doubles. At the same time, original master files are lost every day, leaving behind only downressed and compressed production assets. Even assets produced today through scanning can suffer from blur and noise artifacts. As the definition states, Enhance takes assets and improves the quality in some way, extending their value and lifespan. One ArtEngine feature from the Enhance family is Upres which can perform a 2x or a 4x cartoon or photorealistic Upres on any image or texture.
Going from a concept to filling an entire world is a long, tedious and largely repetitive process. Expansion was the first problem Artomatix set out to solve and it’s still one of our core goals. We’re proud to support a number of Expansion based features within ArtEngine today. Seam Removal can make any texture or material tileable. Content Aware Fill can fill in any damaged or missing parts of a scan. Mutation can grow a material out to a bigger size without creating repeats. It can also generate infinite seamless variants from just a single example. Expansion brings the richness and variety of the real world into the digital realm.
Often 3D content is unfit for the task at hand, not that there’s anything wrong with the content itself, it just doesn’t work with the rest of the project. Perhaps a material is in the wrong artistic style, or perhaps a PBR scan just doesn’t look right placed next to other scans due to scanner miscalibration. If putting together a 3D world is like assembling a puzzle, Transmutation is any function that helps the artist when those puzzle pieces don’t fit together other out of the box. Many real world objects such as cloth and brick walls are “near-regular patterns”, this means they have a pattern, but it’s warped or non-exact in some way and can’t be used as a tileable texture. The Pattern Unwarp feature automatically detects the true pattern and unwarps it, transmuting it into a form that can be tiled.
We’ve briefly introduced the three pillars of automation, or the three ways artists will use an Example-Based workflow to transform their 3D content in the future, making it better, making as much as they could ever want, in whatever form they want it to be.
If you would like to trial ArtEngine, we would love to hear from you. You can sign up here to speak with one of our team about setting up a 2-week free trial. We also have some very exciting roles open at the moment such as a Product Evangelist (perfect for an artist).
Whether it’s doing a trial, applying for a job, engaging with us on social media, reading our newsletter, or saying hello at a conference; I would love to keep in touch!
Dr. Eric Risser
CTO & Founder