T.J. TelanPractical DevOps & Developer ExperienceZola2023-02-04T00:00:00+00:00https://tjtelan.com/atom.xmlJanuary 2023 recap2023-01-31T00:00:00+00:002023-02-04T00:00:00+00:00https://tjtelan.com/blog/end-of-month-summary/<p>This is the first of what will be a series of end of month summaries. Structure won’t always be the same. I’m figuring things out as I go.</p>
<h3 id="professional">Professional<a class="zola-anchor" href="#professional" aria-label="Anchor link for: professional">🔗</a></h3>
<p>Continuing efforts from December into January, I got to sharpen my design skills. Long story short, we were contracting with a couple of UX designers who were producing wireframes, our engagement was ending. But the team benefitted a lot from having something visual to facilitate communicating between the business or developers. </p>
<p>I really liked it. It was a nice change from writing Rust code, and I felt that the effort was very impactful. I don’t think I’ll be going full time into doing any proper UX design, but this reinforced how I feel about the importance of communicating effectively in this software development environment.</p>
<p>As someone who works from home, asynchronous, written communication (with or without visual aid) is more impactful than only communicating face to face. Knowing how to write code has been way less important for me than being capable of expressing ideas or strategy in plain language.</p>
<h3 id="personal">Personal<a class="zola-anchor" href="#personal" aria-label="Anchor link for: personal">🔗</a></h3>
<p>Every January, my partner and I participate in vegan January (aka Veganuary). I like the contrast from the family and food heavy holidays, and the routine of starting every year with this self-imposed restriction. </p>
<p>This is the 7th year we’ve done it, and for the most part it is a non-event. We’re decent home cooks and already eat a mostly plant-based diet, (but I did miss eating eggs for breakfast). As a highlight, I’ve been learning to cook more Filipino food, and most successful experiment was the tofu + potato adobo.</p>
<hr />
<p>My theme for 2023 is improving work-life balance. I’ve been working from home since 2020, and I live with unbalance more towards work. It was already unsustainable, so the intent is taking steps to correct it.</p>
<p>With inspiration from CGP Grey’s <a rel="noopener nofollow" target="_blank" href="https://www.youtube.com/watch?v=snAhsXyO3Ck">Spaceship You</a> and the intro ritual from every Mr. Rogers Neighborhood episodes I’ve seen, I started creating physical cues between my work-space and the rest of the house.</p>
<p>Improvements:</p>
<ol>
<li>Work must be at my desk (with some exceptions for the winter, because my office is in my cold basement), and I have dedicated “work glasses”.</li>
<li>I have a proper “hobby space” in my garage that doesn’t primarily require sitting at a keyboard and monitor. This required a larger cleaning effort, which alone has improved my quality of life.</li>
<li>I have dedicated “life glasses”. No day-job work wearing these glasses. No exceptions!</li>
</ol>
<h3 id="community">Community<a class="zola-anchor" href="#community" aria-label="Anchor link for: community">🔗</a></h3>
<p>A goal is to participate more at my local makerspace. This month I attended a couple events for learning how to use a sewing machine, an embroidery machine, and learning how to use Unity for game making.</p>
<p>It kind of feels bad to not immediately have these skills click. A quote from Adventure Time gives me comfort. “Sucking is the first step to being good at something”. My hope for writing this experience down is to create a little accountability to myself to not quit.</p>
Goals for 20232022-12-30T00:00:00+00:002022-12-31T00:00:00+00:00https://tjtelan.com/blog/goals-2023/<p>Hey everyone,</p>
<p>The year is almost up, and I didn’t want to close the entire year with silence. 2022 did not go as originally planned. I had a major loss in the family, and I didn’t really have the energy to create or share anything this year. I’m feeling ready to restart my plans again. But I’m feeling a little boxed in by restrictions of my own creation, and so I’m going to remove them. Next year will be dedicated to experimenting with the projects and content I work on.</p>
<p>I was reminded that I create publicly because I don’t see or hear people like me in my spaces. The intent behind why I continue to post is to improve my communication and story telling skills. Creating content forces me to better understand what I’m talking about. I’m glad it also supporting others, leading to a collectively better understanding of a topic.</p>
<p>Boundaries on content are being established for 2023. I’ll still write about Rust or DevOps stuff (and a couple posts are in in progress too). I’m not committing to complete them in any specific timeframe, however. They are lower priority, so I can explore ideas outside of my day job. This is to reinforce stricter boundaries with efforts and concepts related to actively solving problems in my day job. Preferably, Day job-type content will be supported during day job hours.</p>
<p>There will be more update-type posts to document my progress on various topics in the short-term. I’m hoping to post more regularly in exchange of spending less effort planning and editing content. Hopefully setting the quality bar expectations lower will take a lot of the pressure off of completing something before posting, because this is supposed to be fun for me. Not an extension for work.</p>
<p>I don’t know how to end this so I’ll just keep it simple. Ranges of topics I write about will expand to outside of tech (but may be adjacent, occasionally). Plans will also include more than written content, as I’m trying to improve my video editing skills too.</p>
<p>Here’s to the new year, 2023. I hope it’s a better one.</p>
How to Build a Custom Integration Test Harness in Rust2021-04-26T00:00:00+00:002021-04-27T00:00:00+00:00https://tjtelan.com/blog/rust-custom-test-harness/<blockquote>
<p>This is a post I wrote for <a rel="noopener nofollow" target="_blank" href="https://www.fluvio.io/blog/2021/04/rust-custom-test-harness/">Fluvio's blog</a></p>
<p>Check out <a rel="noopener nofollow" target="_blank" href="https://github.com/infinyon/fluvio">Fluvio on Github</a></p>
<p>I've posted here for archival purposes.</p>
</blockquote>
<div class="blog-image">
<img src="[object]" alt="The Rust logo plus the Ferris the crab holding a screwdriver and hammer" />
</div>
<p>I ran into a problem effectively using <code>cargo test</code> in <a rel="noopener nofollow" target="_blank" href="https://github.com/infinyon/fluvio">Fluvio</a> for integration testing.</p>
<h2 id="let-s-talk-about-integration-testing-in-rust">Let’s talk about integration testing in Rust<a class="zola-anchor" href="#let-s-talk-about-integration-testing-in-rust" aria-label="Anchor link for: let-s-talk-about-integration-testing-in-rust">🔗</a></h2>
<p>While creating integration testing for Fluvio, I ran into a problem. Organizing and executing integration tests with <code>cargo test</code> was becoming inefficient. We needed to standardize the setup of a test environment. </p>
<p>As a lone developer, you can apply one-off customizations when running tests locally. But if you try to extend that strategy to continuous integration, you’ll quickly find that making changes manually becomes burdensome. CI encourages testing many different configurations, which means a successful CI plan requires easy management of test harness variables (a.k.a. Not manually updating variables for every test you need to run).</p>
<p><code>cargo test</code> is just not equipped to handle this specialized focus on environment setup, or the cleanup/teardown needed after a test is run. When using <code>cargo test</code>, these crucial tasks could only occur outside of the harness or within the logic of a test. Neither of these are good choices. Outside of the harness is not ideal because these processes end up too disconnected and hard to maintain. Likewise, including setup/teardown within the logic of a test is inappropriate because it creates mental overhead for a test writer, and may obscure the results of tests.</p>
<p>I needed to find a way around the limited functionality of <code>cargo test</code> -- keep reading to find out how I did it by creating a standardized setup and teardown as part of our testing harness.</p>
<h3 id="how-does-cargo-test-work-by-default">How does <code>cargo test</code> work by default?<a class="zola-anchor" href="#how-does-cargo-test-work-by-default" aria-label="Anchor link for: how-does-cargo-test-work-by-default">🔗</a></h3>
<p>There is a distinction between unit tests and integration tests in <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/ch11-03-test-organization.html">the Rust book</a>. The distinction is less about testing strategy and more about defining Rust’s conventions for test organization.</p>
<p>The main points are that:</p>
<ul>
<li>Your tests are annotated with <code>#[test]</code></li>
<li><a rel="noopener nofollow" target="_blank" href="https://github.com/rust-lang/libtest">libtest</a> harness enumerates through all of your tests (a point we’ll revisit later in more detail)</li>
<li>libtest returns the pass/fail status of the execution</li>
</ul>
<h3 id="what-do-i-need-from-integration-testing">What do I need from integration testing?<a class="zola-anchor" href="#what-do-i-need-from-integration-testing" aria-label="Anchor link for: what-do-i-need-from-integration-testing">🔗</a></h3>
<p>Libtest doesn't specifically offer anything to support integration testing patterns.</p>
<p>Setup of a standard test environment – especially in a complex system – is essential for managing expected behavior when making code changes.</p>
<p>Unfortunately libtest does not assist with setup or teardown. I needed the ability to abstract away the setup and teardown of my test environment from test code. </p>
<p>This task will be performed either way. Without harness support, setup/teardown will be performed via external shell scripts or padding the setup/teardown process within every single integration test... (no one's idea of fun).</p>
<p>It isn’t convenient to manage setup and teardown in a different context than the integration test. This kind of testing overhead leads to hard-to-reproduce and time consuming mistakes.</p>
<h3 id="where-do-we-get-started-with-a-custom-test-harness">Where do we get started with a custom test harness?<a class="zola-anchor" href="#where-do-we-get-started-with-a-custom-test-harness" aria-label="Anchor link for: where-do-we-get-started-with-a-custom-test-harness">🔗</a></h3>
<p>By default, libtest will compile each of your <code>#[test]</code> labeled functions into their own binary crates (with its own <code>main()</code>) and executes it as part of the test. But we’re going to build all our integration tests into a single crate. This is recommended in order to speed up compile time ([<a rel="noopener nofollow" target="_blank" href="https://endler.dev/2020/rust-compile-times/#combine-all-integration-tests-in-a-single-binary">1</a>], [<a rel="noopener nofollow" target="_blank" href="https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html">2</a>])</p>
<p>First we’re going to create an integration test directory at the root of the crate where we’re going to build our integration test focused binary.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ mkdir integration
</span><span>$ touch integration/main.rs
</span><span>
</span><span style="color:#608b4e;"># Then create a main() function in main.rs
</span></code></pre>
<p>In your Cargo.toml, you want to add </p>
<pre data-lang="toml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-toml "><code class="language-toml" data-lang="toml"><span style="color:#608b4e;"># Cargo.toml
</span><span>
</span><span style="color:#608b4e;"># We'll revisit the `inventory` crate later in the post
</span><span>[</span><span style="color:#808080;">dev-dependencies</span><span>]
</span><span style="color:#569cd6;">inventory </span><span>= </span><span style="color:#d69d85;">"0.1"
</span><span>
</span><span>[[</span><span style="color:#808080;">test</span><span>]]
</span><span style="color:#569cd6;">name </span><span>= </span><span style="color:#d69d85;">"integration"
</span><span style="color:#569cd6;">path </span><span>= </span><span style="color:#d69d85;">"integration/main.rs"
</span><span style="color:#569cd6;">harness </span><span>= </span><span style="color:#569cd6;">false
</span></code></pre>
<p>This tells cargo test to not use libtest when running the <code>integration</code> test.</p>
<p>When we run <code>cargo test integration</code>, what cargo will compile <code>integration/main.rs</code> and execute it in the same manner as <code>cargo run</code>. This is all a harness is from <code>cargo</code>’s perspective. </p>
<h3 id="add-setup-and-teardown-steps">Add Setup and teardown steps<a class="zola-anchor" href="#add-setup-and-teardown-steps" aria-label="Anchor link for: add-setup-and-teardown-steps">🔗</a></h3>
<p>Next we’ll lay the foundation for our testing pattern. We’ll create 2 functions, <code>setup()</code> and <code>teardown()</code>, and add them to our <code>main()</code> (with reserved space in between for our future tests to be called).</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// main.rs
</span><span>
</span><span style="color:#569cd6;">fn </span><span>setup() {
</span><span> println!(</span><span style="color:#d69d85;">"Setup"</span><span>)
</span><span>}
</span><span>
</span><span style="color:#569cd6;">fn </span><span>teardown() {
</span><span> println!(</span><span style="color:#d69d85;">"Teardown"</span><span>)
</span><span>}
</span><span style="color:#569cd6;">fn </span><span>main() {
</span><span> </span><span style="color:#608b4e;">// Setup test environment
</span><span> setup();
</span><span>
</span><span> </span><span style="color:#608b4e;">// TODO: Run the test
</span><span>
</span><span> </span><span style="color:#608b4e;">// Teardown test environment
</span><span> teardown();
</span><span>}
</span></code></pre>
<h3 id="collect-all-integration-tests">Collect all integration tests<a class="zola-anchor" href="#collect-all-integration-tests" aria-label="Anchor link for: collect-all-integration-tests">🔗</a></h3>
<p>To do its job, our test runner needs to create a list of all the test functions. Initially, I thought there would be an easy way to do this by leveraging <a rel="noopener nofollow" target="_blank" href="https://github.com/rust-lang/libtest">libtest</a>'s <code>#[test]</code> attribute. </p>
<p>I dug around in <a rel="noopener nofollow" target="_blank" href="https://github.com/rust-lang/libtest/blob/master/libtest/lib.rs">relevant areas of libtest</a> and <a rel="noopener nofollow" target="_blank" href="https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/cargo_test.rs">Cargo test</a> and <a rel="noopener nofollow" target="_blank" href="https://github.com/rust-lang/rust/blob/master/compiler/rustc_builtin_macros/src/test.rs">Rustc macros</a> code, but long (sad) story short, there is no straightforward way to reuse libtest for the purpose of test collection.</p>
<p>If that surprises you, then you're like me. I had hoped to use the test collection functionality from <code>#[test]</code>, but it wasn’t clear how I could accomplish this. My mental model for how <code>cargo test</code> works needed a refresh.</p>
<p>Now that we’ve removed the option of using libtest, so that gives you 2 practical options for collecting tests:</p>
<ol>
<li>Manually modify <code>integration/main.rs</code> and add your test in between the setup and teardown
<ul>
<li>A quick and straightforward solution if you have a small set of tests</li>
<li>This option requires us to add new tests to this list, which can be error-prone and tedious as we grow.</li>
</ul>
</li>
<li>Build a test collector. We generate an external test catalog, and modify <code>integration/main.rs</code> to execute tests from the catalog.
<ul>
<li>This is a long term solution, which we’ll be covering for the rest of the post.</li>
</ul>
</li>
</ol>
<h3 id="building-the-test-collector">Building the test collector<a class="zola-anchor" href="#building-the-test-collector" aria-label="Anchor link for: building-the-test-collector">🔗</a></h3>
<p>For this test collector, we'll be utilizing a crate. The <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/inventory">inventory</a> crate is a plugin registry system. We'll be using it for all the heavy lifting in our test framework, which means we'll be treating our tests as plugins.</p>
<p>In our <code>main.rs</code>, let’s declare a new module <code>tests</code>, where we can organize all the integration tests.</p>
<pre data-lang="diff" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-diff "><code class="language-diff" data-lang="diff"><span>// main.rs
</span><span>
</span><span>+ pub mod tests;
</span><span>
</span><span>fn setup() {
</span><span> println!("Setup")
</span><span>}
</span><span>
</span><span>fn teardown() {
</span><span> println!("Teardown")
</span><span>}
</span><span>fn main() {
</span><span> // Setup test environment
</span><span> setup();
</span><span>
</span><span> // TODO: Run the test
</span><span>
</span><span> // Teardown test environment
</span><span> teardown();
</span><span>}
</span></code></pre>
<p>In our new module, we’ll start by creating a struct to represent a single test for the plugin registry.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// tests/mod.rs
</span><span>
</span><span>#[derive(Debug)]
</span><span style="color:#569cd6;">pub struct </span><span>IntegrationTest {
</span><span> </span><span style="color:#569cd6;">pub </span><span>name: </span><span style="color:#569cd6;">&'static str</span><span>,
</span><span> </span><span style="color:#569cd6;">pub </span><span>test_fn: </span><span style="color:#569cd6;">fn</span><span>(),
</span><span>}
</span><span>
</span><span>inventory::collect</span><span style="color:#569cd6;">!</span><span>(IntegrationTest);
</span></code></pre>
<p>In this example, our struct <code>IntegrationTest</code> has 2 fields. </p>
<ul>
<li><code>name</code> is a human-readable name, which can be used as a key for test selection.</li>
<li><code>test_fn</code> is a pointer to a function whose signature is non-async, takes no args, and does not return anything.</li>
</ul>
<p>Note: You can use functions that take args, and return things.</p>
<p>For example:</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub</span><span> test_fn: </span><span style="color:#569cd6;">fn</span><span>(String) -> Result<(), ()>,
</span></code></pre>
<p>Then we call the <code>inventory::collect!()</code>macro to instantiate a plugin registry. When we write our tests, we’ll add to the plugin registry. More on this next.</p>
<h3 id="adding-new-tests-to-plugin-registry">Adding new tests to plugin registry<a class="zola-anchor" href="#adding-new-tests-to-plugin-registry" aria-label="Anchor link for: adding-new-tests-to-plugin-registry">🔗</a></h3>
<p>We’re going to add a new basic test to the plugin registry. Start by adding a new submodule called <code>basic</code> in the tests module. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// tests/mod.rs
</span><span>
</span><span style="color:#569cd6;">pub mod </span><span>basic;
</span></code></pre>
<p>In the <code>basic</code> module, we write our basic test <code>basic_test()</code></p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// tests/basic.rs
</span><span>
</span><span style="color:#569cd6;">use super</span><span>::IntegrationTest;
</span><span>
</span><span style="color:#569cd6;">fn </span><span>basic_test() {
</span><span> println!(</span><span style="color:#d69d85;">"Running basic test"</span><span>)
</span><span>}
</span><span>
</span><span>inventory::submit</span><span style="color:#569cd6;">!</span><span>(IntegrationTest {
</span><span> name: </span><span style="color:#d69d85;">"basic"</span><span>,
</span><span> test_fn: basic_test
</span><span>});
</span></code></pre>
<p>We'll use <code>inventory::submit!()</code> to register our new test with the <code>IntegrationTest</code> struct we defined earlier.</p>
<p><code>name</code> is a friendly, human-readable name. We can use this name as a key to search through the plugin registry.</p>
<p><code>test_fn</code> takes the name of our test function. It has the same function signature as we defined. </p>
<h3 id="running-tests-from-registry">Running tests from registry<a class="zola-anchor" href="#running-tests-from-registry" aria-label="Anchor link for: running-tests-from-registry">🔗</a></h3>
<p>We’ll finish this example up by running all of our registered tests</p>
<pre data-lang="diff" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-diff "><code class="language-diff" data-lang="diff"><span>// main.rs
</span><span>
</span><span>pub mod tests;
</span><span>+ use tests::IntegrationTest;
</span><span>
</span><span>fn setup() {
</span><span> println!("Setup")
</span><span>}
</span><span>
</span><span>fn teardown() {
</span><span> println!("Teardown")
</span><span>}
</span><span>fn main() {
</span><span> setup();
</span><span>
</span><span>- // TODO: Run the test
</span><span>+ // Run the tests
</span><span>+ for t in inventory::iter::<IntegrationTest> {
</span><span>+ (t.test_fn)()
</span><span>+ }
</span><span>
</span><span> teardown();
</span><span>}
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo test integration
</span><span> Compiling blog-post-example v0.1.0 (/home/telant/Documents/blog-post-example)
</span><span> Finished test </span><span style="color:#569cd6;">[</span><span>unoptimized + debuginfo</span><span style="color:#569cd6;">]</span><span> target(s) in 0.21s
</span><span> Running target/debug/deps/blog_post_example-e042d787684bb333
</span><span>
</span><span>running 0 tests
</span><span>
</span><span>test result: ok. 0 passed</span><span style="color:#569cd6;">; </span><span>0 failed</span><span style="color:#569cd6;">; </span><span>0 ignored</span><span style="color:#569cd6;">; </span><span>0 measured</span><span style="color:#569cd6;">; </span><span>0 filtered out</span><span style="color:#569cd6;">; </span><span>finished in 0.00s
</span><span>
</span><span> Running target/debug/deps/integration-7ed2452642c6f3b6
</span><span>
</span><span>Setup
</span><span>Running basic test
</span><span>Teardown
</span></code></pre>
<h3 id="tips-for-extending-the-example">Tips for extending the example<a class="zola-anchor" href="#tips-for-extending-the-example" aria-label="Anchor link for: tips-for-extending-the-example">🔗</a></h3>
<p>The example runs all of the registered tests. But here are some useful impls if you want to extend even further. For example, adding a CLI, if you want to select individual tests. Or provide options to customize setup or teardown behavior.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">impl </span><span>IntegrationTest {
</span><span> </span><span style="color:#569cd6;">pub fn </span><span>all_test_names() -> Vec<</span><span style="color:#569cd6;">&'static str</span><span>> {
</span><span> inventory::iter::<IntegrationTest>
</span><span> .into_iter()
</span><span> .map(|x| x.name)
</span><span> .collect::<Vec<</span><span style="color:#569cd6;">&str</span><span>>>()
</span><span> }
</span><span>
</span><span> </span><span style="color:#569cd6;">pub fn </span><span>from_name<S: AsRef<</span><span style="color:#569cd6;">str</span><span>>>(test_name: S) -> Option<</span><span style="color:#569cd6;">&'static</span><span> IntegrationTest> {
</span><span> inventory::iter::<IntegrationTest>
</span><span> .into_iter()
</span><span> .find(|t| t.name == test_name.as_ref())
</span><span> }
</span><span>}
</span></code></pre>
<blockquote>
<p>If you want to see more of these ideas extended even further, check out <a rel="noopener nofollow" target="_blank" href="https://github.com/infinyon/fluvio/tree/master/tests/runner/src">Fluvio’s integration test runner</a>.</p>
<p>We use the CLI to customize setup, handle async testing, and we use an <a rel="noopener nofollow" target="_blank" href="https://github.com/infinyon/fluvio/blob/master/tests/runner/src/fluvio-integration-derive/src/lib.rs">attribute macro</a> to collect tests.</p>
</blockquote>
<h3 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h3>
<p>Rust’s testing ecosystem in the 2018 edition is great for unit testing. But for integration testing it still has room for improvement. Custom harnesses will become more necessary as Rust gains new developers. </p>
<p>If we want to avoid reinventing the wheel, we need stable support from <a rel="noopener nofollow" target="_blank" href="https://docs.rs/libtest/0.0.1/libtest/">libtest</a> or more examples of how to perform test collection and patterns for setup, test, and teardown workflows.</p>
<p>If you made it this far, thank you for following along with me! I wrote this because I could not find a guide to do this before trying to do this myself, and knowing these things beforehand would have made it much faster. Hopefully others find my experience helpful.</p>
Back from my writing break2021-04-26T00:00:00+00:002021-04-26T00:00:00+00:00https://tjtelan.com/now/<p>I recently joined <a rel="noopener nofollow" target="_blank" href="https://www.infinyon.com/">Infinyon</a> to work on <a rel="noopener nofollow" target="_blank" href="https://www.fluvio.io/">Fluvio</a>, a data-streaming platform written in Rust.</p>
<p>My focus is on process improvements and performance type testing. It is giving me a lot of opportunity to learn more about Rust.</p>
<p>I have plans on venture out into making video educational content in the next quarter or so. Hopefully more on that subject soon.</p>
9 Insights I Uncovered While Building a Writing Habit2021-01-28T00:00:00+00:002021-01-28T00:00:00+00:00https://tjtelan.com/blog/9-insights-i-uncovered-while-building-writing-habit/<h2 id="building-a-writing-habit">Building a writing habit<a class="zola-anchor" href="#building-a-writing-habit" aria-label="Anchor link for: building-a-writing-habit">🔗</a></h2>
<p>In the forever-trying year of 2020, I set my sights on earning the <a rel="noopener nofollow" target="_blank" href="https://dev.to/badge/16-week-streak">16 week posting streak badge on Dev.to</a> – and <a rel="noopener nofollow" target="_blank" href="https://dev.to/tjtelan">I did it</a>! (Then I took a long break from posting.) Before I get started on more tech writing for 2021, I wanted to stop and reflect on this effort.</p>
<h3 id="how-did-it-go">How did it go?<a class="zola-anchor" href="#how-did-it-go" aria-label="Anchor link for: how-did-it-go">🔗</a></h3>
<p>Pretty well, to be honest. It has been one of the most challenging goals I’ve set for myself. After the 50% mark, I had exhausted the developed ideas in my backlog so it became harder to get in front of my schedule. I had to start the next post immediately after publishing.</p>
<p>I found a few insights while building my habit for writing, and you might find them helpful, too.</p>
<h2 id="1-the-hardest-part-is-getting-started">1. The hardest part is getting started<a class="zola-anchor" href="#1-the-hardest-part-is-getting-started" aria-label="Anchor link for: 1-the-hardest-part-is-getting-started">🔗</a></h2>
<p>It’s easy to say that some of this is due to the mood of 2020, but it was really easy to fall into self-doubt and self-rejection based on the worry that no one would read what I was going to write. Or that people would read it, but they’d hate it.</p>
<p>These are still feelings that I sometimes have today. But aside from SEO, marketing on social media, experimenting with the time I post, and other tasks that take effort – whether or not someone clicks and reads is out of my control.</p>
<p>Due to the mental effort required to overcome these feelings, the hardest part of writing is getting started. As such, it (almost) doesn’t matter what the resulting quality is. Primarily what matters is getting something on one topic on the page, to edit later. </p>
<h2 id="2-editing-is-easier-than-writing">2. Editing is easier than writing<a class="zola-anchor" href="#2-editing-is-easier-than-writing" aria-label="Anchor link for: 2-editing-is-easier-than-writing">🔗</a></h2>
<p>I have a much easier time editing than writing from scratch. Knowing this about myself meant that it was a lot more important to complete any kind of first draft, no matter how bad.</p>
<p>The editing phase takes at least twice as long as writing. This phase may involve complete rewrites of sections, but I still consider that to be less difficult than starting from a blank page. It’s also a lot easier to judge how much I have left to do when the edits are focused on formatting and pacing, and less on content or coherency. </p>
<h2 id="3-don-t-be-a-perfectionist">3. Don’t be a perfectionist<a class="zola-anchor" href="#3-don-t-be-a-perfectionist" aria-label="Anchor link for: 3-don-t-be-a-perfectionist">🔗</a></h2>
<p>At least for the content that I’m creating, most people aren’t going to suddenly understand a complicated concept because I wrote a specific sentence in a particular way.</p>
<p>For the goal I was working toward, it was important to post on a predictable schedule. While it’s good to set your own standards for yourself high, keep in mind that it’s impossible to level up 10x with a single post. Writing well takes practice, and you get more practice if you are able to let go, call a story done, and move on to writing something else.</p>
<h2 id="4-writing-consistently-doesn-t-always-yield-consistent-quality">4. Writing consistently doesn’t always yield consistent quality<a class="zola-anchor" href="#4-writing-consistently-doesn-t-always-yield-consistent-quality" aria-label="Anchor link for: 4-writing-consistently-doesn-t-always-yield-consistent-quality">🔗</a></h2>
<p>I’ll be the first to admit that I think some of my posts are more useful or interesting to read than others. That’s just how it goes sometimes, where ambitions are better than the content produced.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Ira_Glass">Ira Glass</a> spoke about storytelling and <a rel="noopener nofollow" target="_blank" href="https://www.youtube.com/watch?v=X2wLP0izeJE">the taste gap (and how to close it</a>). He encourages that the most important thing one can do when starting out is to produce quantity over quality. Over time you improve, if you can endure this creative, iterative phase. It is normal for quality to lag behind your ambitions. </p>
<p>With that in mind, it’s okay to write things that you never publish. I’ve learned that writing can be a method for understanding a topic, or just organizing my thoughts. Not all of that needs to be public.</p>
<p>What continued to give me encouragement was noticing that my first draft quality was (on average) getting better over time. That was a measurable improvement! </p>
<h2 id="5-be-really-specific-about-who-you-are-talking-to">5. Be really specific about who you are talking to<a class="zola-anchor" href="#5-be-really-specific-about-who-you-are-talking-to" aria-label="Anchor link for: 5-be-really-specific-about-who-you-are-talking-to">🔗</a></h2>
<p>When I write, I always start the first draft always with a question: Who am I writing to?</p>
<p>It doesn’t always matter if the answer is a real person. This person I’m writing to might be based on my past experience, questions people have asked me, interactions I’ve observed, an organization, or just completely made up!</p>
<p>What is most important is that I focus my message so my intended target audience understands. I don’t want to unintentionally expand the audience or over-explain a concept if it doesn’t benefit the base group of people I am writing to. This technique has made it a lot easier to write in bursts, and edit for clarity. It helps each piece of writing have a consistent voice.</p>
<h2 id="6-writing-titles-is-hard-because-seo-is-hard">6. Writing titles is hard (because SEO is hard)<a class="zola-anchor" href="#6-writing-titles-is-hard-because-seo-is-hard" aria-label="Anchor link for: 6-writing-titles-is-hard-because-seo-is-hard">🔗</a></h2>
<p>I have to admit: I change the title of each post no less than 3 times, sometimes minutes before publishing the post!</p>
<p>Different publishing platforms have different audiences, so experiment with what works for your readers. If you plan on cross-posting across multiple platforms or you want to be discovered through search engines, you will need to learn a little bit about marketing and <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Search_engine_optimization">SEO</a>. </p>
<p>I’m frequently torn between using technical jargon in my titles, or less specific concepts that can be generally applied.</p>
<p>Playing with this balance is a cheap and interesting game that, in my experience, avoids disappointment by taking a long-term perspective. It’s been validating to see what kind of content does well though organic search visits vs. people sharing links. I’m still caught off guard when an article from many months ago begins to trend. The lesson here is to be patient and see what strategies pay off.</p>
<h2 id="7-keep-a-backlog-of-ideas">7. Keep a backlog of ideas<a class="zola-anchor" href="#7-keep-a-backlog-of-ideas" aria-label="Anchor link for: 7-keep-a-backlog-of-ideas">🔗</a></h2>
<p>I have a spreadsheet that I use to keep track of ideas in various states. When I think of something that might be a good topic to explore in a post, I add it to the list. These can be title ideas, posts that are in progress, and links to drafts. </p>
<p>I try not to dismiss an idea too early. Just writing it down gets it out of my mind, with low pressure to fully explore the idea. Sometimes this collection of thoughts ends up presenting patterns or themes that, when combined, contain enough material to create a full post around. This strategy is really practical, and comes in handy when I am feeling like I’m in a creative rut. If I have a deadline to meet, I can always pull from the pile of ideas I already had.</p>
<h2 id="8-building-structure-and-motivation-for-writing-is-moving-target">8. Building structure and motivation for writing is moving target<a class="zola-anchor" href="#8-building-structure-and-motivation-for-writing-is-moving-target" aria-label="Anchor link for: 8-building-structure-and-motivation-for-writing-is-moving-target">🔗</a></h2>
<p>I am generally informed about trending topics within tech, but I’m trying to discover my niche and I don’t want to spend an eternity looking for ideas. I’ve also discovered that I’m most creative when I have constraints and have a structure to lean on.</p>
<p>I have a long-term goal to work for myself creating educational products / services / consulting. So a structure I lean into is that each post contributes to that long-term goal. </p>
<p>It’s also important to address shorter-term motivation. My approach has been to identify what motivates me to create, and then slowly build systems to support producing content. Aside from the obvious need to set aside time to write, creating that motivation for a post takes three things: </p>
<ul>
<li>The content has to be something that is <strong>not too difficult</strong> for me to create. This means I mostly know the material I’m talking about, and I’m not learning new skills specifically for the post. (It’s not uncommon for me to write about a learning experience, though! I just don’t set out to learn how to do something specifically to write a post)</li>
<li><strong>Something I enjoy creating</strong> - The topic I’m writing about needs to be interesting, or the story I’m telling needs to be meaningful. Otherwise, I’ll get bored writing it (or you’ll get bored reading it).</li>
<li>The result should <strong>provide value to someone else</strong>. People value their time, so in writing an article, I’m trading my time writing for someone’s time savings. What I communicate can save people time, and that’s meaningful to me. I know I have benefitted from others doing the same.</li>
</ul>
<p>I really enjoy creating software developer advocacy and tech education content, and the topics I (mostly) write on reinforce these areas.</p>
<p>I aspire to have objectives that can be described as <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/SMART_criteria">SMART</a>. Admittedly, my short- and long-term goals are still pretty abstract. I think that’s okay right now, as long as I am making progress in the right direction.</p>
<h2 id="9-don-t-be-afraid-to-be-seen-trying">9. Don’t be afraid to be seen trying<a class="zola-anchor" href="#9-don-t-be-afraid-to-be-seen-trying" aria-label="Anchor link for: 9-don-t-be-afraid-to-be-seen-trying">🔗</a></h2>
<p>Credit to a video by <a rel="noopener nofollow" target="_blank" href="https://www.youtube.com/watch?v=hFQRx5iwLtw">Evelyn From The Internets</a> for helping me articulate my feelings about this point.</p>
<p>This relates to my previous point of starting being the hardest part. Some of my internal friction stems from the feeling of being afraid to be seen trying something I’m not good at.</p>
<p>With some rare exceptions, I’ve decided to keep my old content online. It likely won’t be an issue where stuff I created in the past while I was still learning has embarrassed me. Largely, this content is ignored and no one but myself cares about it. </p>
<p>To be concise, haters gonna hate. That’s what they do. So don’t pay any attention to the inconsiderate folks who “just want to help” but are not actually helpful. The tech community is full of strong personalities that aren’t always inviting or pleasant to engage with. (I’m not saying this is the majority. I see you...) </p>
<p>I see enough 💩 being talked in social media threads that it constantly occupies space in my mind when I’m about to publish a new piece of content. However, the truth is that most people are not focused on you for even a moment. So a way I like to view that is that most people will not even see you fail – that’s the blessing of no one knowing who you are yet. Therefore, there’s no reason to not try.</p>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>Building a writing habit has been very humbling. I learned that I actually know a lot less about certain topics than I thought. I had to research more than I expected. But as I gained experience, I leveled up my time management and my process.</p>
<p>Now that I’m not trying to post weekly, what I would do differently is to spend more time developing my production processes. And that’s what the focus of this next year will be about.</p>
<p>Aside from continuing to <a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/tjtelan">stream my product development on Twitch</a>, this year I will be creating short videos, starting with some of my more popular blog posts as topics. I hope this will lead into more types of educational content, and more opportunities to work with other people in achieving their goals.</p>
<p>I started a consulting business (<a rel="noopener nofollow" target="_blank" href="https://figbeelabs.com/">Figbee Labs</a>) in the latter half of 2020. I will continue to travel out of my comfort zone into the deeper waters and hopefully meet new people or organizations that want to work together. It’s important to me to help people from all backgrounds succeed in sustainably building and running their own software products. I will document my experience over 2021 through posts and various social media. </p>
<p>If you’ve undertaken a similar challenge, what did you learn about your process? What do you want to achieve with writing this year?</p>
Questions to Find the Right Continuous Integration System for You2020-12-10T00:00:00+00:002020-12-10T00:00:00+00:00https://tjtelan.com/blog/questions-to-find-the-right-continuous-integration-system-for-you/<p>Continuous integration is fancy shell scripting. It even uses shell scripting terminology like pipelines, but is obscured with marketing and many seemingly complex and different methods of configuration.</p>
<p>The purpose of this post is to discuss Continuous Integration (CI) Systems at a high level if you are considering switching to a new service. I want to introduce you to a few talking points that aren't discussed enough in order to help you make a decision.</p>
<h2 id="what-are-the-goals-of-continuous-integration-ci">What are the goals of Continuous Integration (CI)?<a class="zola-anchor" href="#what-are-the-goals-of-continuous-integration-ci" aria-label="Anchor link for: what-are-the-goals-of-continuous-integration-ci">🔗</a></h2>
<p>The functions of continuous integration are very flexible and therefore, and the goals are super simple. Automatically run scripts upon the occurrence of an event.</p>
<p>Common events can be a git commit or a timer event set to run every 5 minutes.</p>
<h2 id="how-does-ci-differ-from-shell-scripting">How does CI differ from shell scripting?<a class="zola-anchor" href="#how-does-ci-differ-from-shell-scripting" aria-label="Anchor link for: how-does-ci-differ-from-shell-scripting">🔗</a></h2>
<p>Continuous integration has a narrow context in which it is intended to run. It has extra structure in order to tell a tighter story. (The CI story leads into another story called Continuous Delivery/Deployment (CD), but we will only be focusing on CI)</p>
<p>A best practice of CI is to automatically build and/or run tests on every commit in order to validate whether the codebase builds/tests are complete without errors.</p>
<p>Basic structure defines conditions as well as commands or actions that should be executed when an event occurs. Conditions such as a new commit in a specific branch, or file paths. Or keywords in commit messages.</p>
<p>On the more advanced side, platform specific features like plugins may be introduced for sending or receiving events from other systems, initiating external triggers based on other pipelines, caching, cleaning up and notifications.</p>
<h2 id="what-type-of-ci-systems-that-exist-today">What type of CI systems that exist today?<a class="zola-anchor" href="#what-type-of-ci-systems-that-exist-today" aria-label="Anchor link for: what-type-of-ci-systems-that-exist-today">🔗</a></h2>
<p>There is no shortage of software that is designed for Continuous Integration. What I believe sets these platforms apart from each other is:</p>
<ul>
<li>The experience and effort involved with setting up a new environment for your organization</li>
<li>Path to configuring new pipelines, and maintaining them over time.</li>
</ul>
<p>If your organization regularly exercises the creation of new environments, like in consulting orgs, it is reasonable to gain the experience with an open source tool that won’t cost you any money with licensing.</p>
<p>Otherwise, I might suggest picking a system the reduces the friction for pipeline creation and maintenance.</p>
<hr />
<p>Styles of setting up new pipelines falls into two categories.</p>
<h3 id="stateful-webui-pipeline-config">Stateful (WebUI) pipeline config<a class="zola-anchor" href="#stateful-webui-pipeline-config" aria-label="Anchor link for: stateful-webui-pipeline-config">🔗</a></h3>
<p>Jenkins is the primary example of configuration via WebUI, as it is the functionality out of the box. Adding pipelines in this system requires a user to log into the WebUI, configure connections to builder systems and methodically define pipelines.</p>
<h4 id="pros">Pros<a class="zola-anchor" href="#pros" aria-label="Anchor link for: pros">🔗</a></h4>
<ul>
<li>The biggest player in this space, Jenkins, is free to install with flexible system requirements.</li>
<li>Relatively low skill required to get started. Setup of pipelines and any subsequent edits are performed through the browser with a few clicks.</li>
<li>Many plugins exist, which probably suit the needs of your org.</li>
<li>Most of the CI in this category are older and mature. Very search engine friendly solutions.</li>
</ul>
<h4 id="cons">Cons<a class="zola-anchor" href="#cons" aria-label="Anchor link for: cons">🔗</a></h4>
<ul>
<li>Administration is more difficult and time consuming to perform as footprint increases. Over time the details of the pipeline configuration may drift and be difficult to reproduce for the purposes of documentation or migration.</li>
<li>It is easy to neglect this kind of CI installation. Both a testament to stability, and a detriment to your org when unexpected outages occur. Rebuilding can be tedious and error prone.</li>
<li>Takes a lot of effort to secure natively in a cloud-based future. Typically these CI systems inconveniently end up hiding behind a VPN.</li>
</ul>
<h3 id="stateless-file-based-pipeline-config">Stateless (File-based) pipeline config<a class="zola-anchor" href="#stateless-file-based-pipeline-config" aria-label="Anchor link for: stateless-file-based-pipeline-config">🔗</a></h3>
<p>This is the modern style of CI system. Your TravisCI, Github Action, Azure DevOps</p>
<p>Typically we find this system is hosted through your git provider, or as a separate hosted service that we allow access to the repos. YAML is the most popular config format. </p>
<h4 id="pros-1">Pros<a class="zola-anchor" href="#pros-1" aria-label="Anchor link for: pros-1">🔗</a></h4>
<ul>
<li>Configuration of pipelines stay versioned alongside the codebases that use them.</li>
</ul>
<h4 id="cons-1">Cons<a class="zola-anchor" href="#cons-1" aria-label="Anchor link for: cons-1">🔗</a></h4>
<ul>
<li>No config schema standards, though. most use a yaml format.</li>
<li>Complexity becomes harder to manage, and requires a commit and push to repos in order to test. Testing contributes clutter in git logs, and uses up build minutes, which is the cost unit that many hosted services bill on.</li>
<li>Limited self-hosted options that are free.</li>
<li>Limited self-hosted options that have small system requirements</li>
</ul>
<h3 id="my-opinion">My opinion<a class="zola-anchor" href="#my-opinion" aria-label="Anchor link for: my-opinion">🔗</a></h3>
<p>I strongly prefer the stateless type of configuration because it provides a more code-like arena where documentation can be provided.</p>
<p>Mixing the workflow of CI pipeline configuration with your organization’s version control practices can be disruptive if you need to push code in order to see the results of changes to your pipeline. It is a price to pay for the ability to easily correlate changes in codebase with changes in pipeline.</p>
<p>The balance to strike rests on your code hosting provider needs (i.e. costs per user) and the types of technologies and platforms you need to support for building (web, native, mobile, embedded, etc). This flexibility mostly skews support towards building for web-based technologies.</p>
<p>Also, Jenkins technically has file-based pipeline support via Jenkinsfile, but unless my org was already deeply invested in staying with Jenkins, I would steer clear. It is not the future. It is harder than necessary to use.</p>
<h2 id="what-are-the-ci-hosting-options">What are the CI Hosting Options?<a class="zola-anchor" href="#what-are-the-ci-hosting-options" aria-label="Anchor link for: what-are-the-ci-hosting-options">🔗</a></h2>
<h3 id="self-hosting">Self-hosting<a class="zola-anchor" href="#self-hosting" aria-label="Anchor link for: self-hosting">🔗</a></h3>
<p>Self-hosting means running and maintaining a process on a physical or virtual machine that you have access to.</p>
<h4 id="pros-2">Pros<a class="zola-anchor" href="#pros-2" aria-label="Anchor link for: pros-2">🔗</a></h4>
<ul>
<li>Somewhat more private for less effort if hosting in a private location.</li>
<li>Effective use of data center computers. Initial costs may be higher, but maintenance costs are typically lower.</li>
</ul>
<h4 id="cons-2">Cons<a class="zola-anchor" href="#cons-2" aria-label="Anchor link for: cons-2">🔗</a></h4>
<ul>
<li>Being responsible for maintenance</li>
<li>Limited to the platforms that you have available.</li>
</ul>
<h3 id="system-as-a-service-hosted">System as a Service (Hosted)<a class="zola-anchor" href="#system-as-a-service-hosted" aria-label="Anchor link for: system-as-a-service-hosted">🔗</a></h3>
<p>External systems as a service (Hosted services) allow users to just log in and connect their version control.</p>
<h4 id="pros-3">Pros<a class="zola-anchor" href="#pros-3" aria-label="Anchor link for: pros-3">🔗</a></h4>
<ul>
<li>Not responsible for performing maintenance.</li>
<li>Typically authentication is provided, with options to support your org’s auth provider</li>
</ul>
<h4 id="cons-3">Cons<a class="zola-anchor" href="#cons-3" aria-label="Anchor link for: cons-3">🔗</a></h4>
<ul>
<li>Can be expensive if you require a lot of build time or resources</li>
<li>Shrinking caps on “free” resources for open source projects</li>
</ul>
<h2 id="issues-to-consider-when-switching-between-ci-platforms">Issues to Consider when Switching between CI platforms<a class="zola-anchor" href="#issues-to-consider-when-switching-between-ci-platforms" aria-label="Anchor link for: issues-to-consider-when-switching-between-ci-platforms">🔗</a></h2>
<p>These are some categorized rhetorical questions that you or your organization should ask if you are migrating CI platforms</p>
<h4 id="threat-model">Threat model<a class="zola-anchor" href="#threat-model" aria-label="Anchor link for: threat-model">🔗</a></h4>
<ul>
<li>Do you need to be deployed on-site?</li>
<li>What are the upfront and ongoing costs associated?</li>
<li>Do you have secrets (like private keys, API tokens) that are needed at build time?</li>
</ul>
<h4 id="contributor-model">Contributor model<a class="zola-anchor" href="#contributor-model" aria-label="Anchor link for: contributor-model">🔗</a></h4>
<ul>
<li>How are contributors logging in?</li>
<li>What are the costs associated per seat?</li>
</ul>
<h4 id="maintenance">Maintenance<a class="zola-anchor" href="#maintenance" aria-label="Anchor link for: maintenance">🔗</a></h4>
<ul>
<li>Who is responsible for the health of the pipeline?</li>
<li>How much time/money are you budgeting to spend on maintenance and improvements?</li>
</ul>
<h4 id="cost-of-build-minutes">Cost of Build minutes<a class="zola-anchor" href="#cost-of-build-minutes" aria-label="Anchor link for: cost-of-build-minutes">🔗</a></h4>
<ul>
<li>For open source projects, how many build minutes are being used?</li>
</ul>
<h4 id="responsiveness-needs">Responsiveness needs<a class="zola-anchor" href="#responsiveness-needs" aria-label="Anchor link for: responsiveness-needs">🔗</a></h4>
<ul>
<li>How many concurrent builds does your org require?</li>
</ul>
<h4 id="technologies-used">Technologies used<a class="zola-anchor" href="#technologies-used" aria-label="Anchor link for: technologies-used">🔗</a></h4>
<ul>
<li>Containerizing builds is popular for isolation. But not all projects can build within a container.</li>
<li>Apple, Microsoft and embedded projects are not nearly as flexible and may require special considerations or resources.</li>
</ul>
<h4 id="delivery-deployment-model">Delivery/Deployment model<a class="zola-anchor" href="#delivery-deployment-model" aria-label="Anchor link for: delivery-deployment-model">🔗</a></h4>
<ul>
<li>What do you need to happen after a build passes?</li>
<li>Are you creating a build artifact that needs to be stored somewhere?</li>
<li>What is the desired frequency between build vs delivery/deploy?</li>
</ul>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>Not all continuous integration is considered equal. But they all provide basically the same functions. In most cases, it is possible to self-host and do everything yourself. But sometimes it is worth paying money to a product / service that takes care of your needs. It is really all about making sure you know what the trade-offs are for your specific case.</p>
Re: Reading More Technical Content Can Help You Improve Your Content Creation Skills2020-12-03T00:00:00+00:002020-12-03T00:00:00+00:00https://tjtelan.com/blog/re-reading-more-technical-content-can-help-you-improve-your-content-creation-skills/<p>This post is in response to <a rel="noopener nofollow" target="_blank" href="https://www.stephaniemorillo.co/post/reading-more-technical-content-can-help-you-improve-your-content-creation-skills-here-s-how">Stephanie Morillo’s post</a></p>
<p>The main piece of advice that is elaborated upon is to <strong>read a lot of technical content</strong>.</p>
<p>What I think is very helpful about this post are the questions that are asked. I thought it would be a good idea to answer these questions so that I can share my opinions around how I read through technical content.</p>
<h2 id="why-do-i-like-a-piece-of-technical-content">Why do I like a piece of technical content?<a class="zola-anchor" href="#why-do-i-like-a-piece-of-technical-content" aria-label="Anchor link for: why-do-i-like-a-piece-of-technical-content">🔗</a></h2>
<p>I can answer this in different ways depending on the context that leads me to find technical content. The main two contexts being discovered through social media, or via a search engine.</p>
<h3 id="through-social-media">Through social media<a class="zola-anchor" href="#through-social-media" aria-label="Anchor link for: through-social-media">🔗</a></h3>
<p>I spend a lot of time on technical social media, like Reddit, Hacker News, Dev.to, and Twitter and I mostly only read the trending posts. So in these cases, an individual’s or company’s personality drives my interest over the actual content.</p>
<p>A lot of the time this leads to some cult of personality for popular posts in the comment section. This can’t be helped. However, this kind of engagement occasionally leads to interesting and tangential discussion. I like when the author can participate in the discussion as well, but this is a rare occurrence.</p>
<p>I prefer concepts over code in the case of social media discovery. Emphasis when I’m reading through something that is about an ecosystem I don’t have experience using. I like having my hand held lightly, and guided through the examples with metaphor or images.</p>
<p>Bonus points if you both provide links for more context, as well as provide a simplified summary. Help frame how I should think, and even give negative examples for how I should not think if it helps.</p>
<p>These posts are primarily to entertain me, and then to inform me. I want the attention to readability with layout and markup to be more considered. If there are images to help set tone, I’d rather they be well used memes, or diagrams.</p>
<h3 id="through-search">Through search<a class="zola-anchor" href="#through-search" aria-label="Anchor link for: through-search">🔗</a></h3>
<p>If I’m searching for something technical, it’s more likely that I’m doing research, design or implementation. So personality may continue to play a role in my likelihood to revisit an article, but the code snippets and coherence of an explanation is more important.</p>
<p>Bonus points if consideration was made for keyword searching or organized textbook-like. I don’t need my hand held as much in his case. Just point me in a direction. I don’t want the entire post to be read through carefully unless the post is about low-level detail.</p>
<p>I’ll be grumpy, but I will endure a post that is harder to skim if the content is promising enough.</p>
<h2 id="who-is-their-intended-audience">Who is their intended audience?<a class="zola-anchor" href="#who-is-their-intended-audience" aria-label="Anchor link for: who-is-their-intended-audience">🔗</a></h2>
<p>Part of reading a lot of technical content means reading things that are not specifically targeted at myself.</p>
<p>When the content involves discussion of code, I really prefer a lot of gentle, but explicit nudging around the skill level required to make sense of what is in the post.</p>
<p>Wide audience is generally my least favorite because content has to be high-level enough to not lose anyone. If you’re aiming wide, then say so or be very obvious. </p>
<p>Textbook-style is good enough for me. Build up concepts in a reasonable sequence. Or link to posts or definitions that talk about foundational concepts that you’re introducing. Unless you’ve already stated an expected skill level, I will feel like you don’t know the audience you’re writing to.</p>
<h2 id="what-does-this-company-individual-do-really-well">What does this (company/individual) do really well?<a class="zola-anchor" href="#what-does-this-company-individual-do-really-well" aria-label="Anchor link for: what-does-this-company-individual-do-really-well">🔗</a></h2>
<p>This kind of question is mostly for judging a collection of posts. Any single post may have left positive impressions on me of being informative, concise, funny, easy to read. But I don’t feel like I can judge what the competencies or patterns of a company or individual is until I see other work.</p>
<p>I look for consistency in voice. Are we having a conversation. Am I reading a monologue? Do you know where your content beats and break things up with whitespace or images? Are you flexible enough with how you present ideas so that I understand you or know what homework I have in order to understand? Visual branding also comes into mind when discussing consistency. </p>
<h2 id="what-does-this-content-help-me-do">What does this content help me do?<a class="zola-anchor" href="#what-does-this-content-help-me-do" aria-label="Anchor link for: what-does-this-content-help-me-do">🔗</a></h2>
<p>My expectations of what the content offers me is again, dependent on the context in which I was introduced.</p>
<p>In general, if it is through social media, I’m looking first to have some kind of entertainment aspect. I’m looking for a breadth of information that allows me to insert my own ideas. I know that content creators aren’t always in control of where and when their content is posted, so this isn’t always going to be fair to judge entertainment value when the content is informational. But it is a plus if the author can be engaging and educational because it takes effort to do well.</p>
<p>Conversely, if it is through organic search, I’m looking for some kind of informational value. Help me solve a problem, or understand a model of thinking better.</p>
<h2 id="how-would-doing-or-implementing-x-improve-my-content">How would doing (or implementing) “X” improve my content?<a class="zola-anchor" href="#how-would-doing-or-implementing-x-improve-my-content" aria-label="Anchor link for: how-would-doing-or-implementing-x-improve-my-content">🔗</a></h2>
<p>This is a great question. Often when I ask myself this question, I’m considering the visual branding of an individual. But lately I’ve also been thinking about how my favorite creators structure and pace the flow of information in their posts while balancing their voice. This is why it is a good idea to follow content creators that resonate with you, because everyone learns to strike this balance differently. And typically the more someone writes technical content, the more natural and easier to read they get.</p>
<h2 id="what-can-they-improve-upon-to-make-this-content-even-more-effective">What can they improve upon to make this content even more effective?<a class="zola-anchor" href="#what-can-they-improve-upon-to-make-this-content-even-more-effective" aria-label="Anchor link for: what-can-they-improve-upon-to-make-this-content-even-more-effective">🔗</a></h2>
<p>At least for me, it is easier to edit someone else’s work. Naturally, I’m constantly thinking about how little bits of friction can be smoothed out and so that I could understand it better. This could mean mixing the styles of creators together, and I try to identify creators who are particularly good (or sometimes bad) at one thing so I can pay attention to how their style evolves.</p>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>I don’t know if this post will be useful to others, but it was a good exercise for me. If you’re a content creator, I encourage you to read through Stephanie’s original post. Ask yourself these questions and try to give concrete answers. </p>
Quick Primer to Practical Reproducible builds: Reference Environments2020-11-27T00:00:00+00:002020-11-27T00:00:00+00:00https://tjtelan.com/blog/quick-primer-reference-environments/<h2 id="overview">Overview<a class="zola-anchor" href="#overview" aria-label="Anchor link for: overview">🔗</a></h2>
<p>This post is inspired by a larger <a rel="noopener nofollow" target="_blank" href="https://twitter.com/bitshiftmask/status/1321623304004866050">Twitter thread by James Munns</a> about industry practices to follow for embedded systems projects that are not covered in school. Particularly the concept of a reference environment. However this idea is important across all platforms of software development. </p>
<p>Our aim is to be pragmatic to a wide audience of “Developers” and not to be confused with the goals of the <a rel="noopener nofollow" target="_blank" href="https://reproducible-builds.org/">reproducible builds project</a>. (I’m not worrying about achieving <a rel="noopener nofollow" target="_blank" href="https://reproducible-builds.org/docs/deterministic-build-systems/">complete build environment determinism</a>. There is a lot more involved that is out of scope for this discussion, but I encourage others to check it out.)</p>
<h2 id="what-s-the-benefit">What’s the benefit?<a class="zola-anchor" href="#what-s-the-benefit" aria-label="Anchor link for: what-s-the-benefit">🔗</a></h2>
<p>The benefit of a reference environment is to allow a practical step-by-step plan for individuals to reproduce a suitable environment in order to build from a codebase, produce artifacts and interface with any external hardware.</p>
<p>I want to help you form a maintainable process of documenting what is needed to build your code, and create artifacts for release.</p>
<p>The expected outcome is to create a reference environment for build and release purposes.</p>
<p>An additional side-effect of this effort is a shortening of the effort to onboard new devs and testers into new projects or codebases. Additionally version control of this process will reduce your organization’s institutional knowledge.</p>
<h3 id="your-reference-environment-can-be-used-for">Your reference environment can be used for:<a class="zola-anchor" href="#your-reference-environment-can-be-used-for" aria-label="Anchor link for: your-reference-environment-can-be-used-for">🔗</a></h3>
<ul>
<li>CI environments</li>
<li>Onboarding new developers</li>
<li>Projects that don’t get modified very often</li>
<li>Embedded projects</li>
<li>Mobile projects</li>
<li>Projects with private code</li>
<li>Projects with vendored / version pinned code</li>
<li>Legacy code</li>
</ul>
<h2 id="start-your-reference-environment-in-steps">Start your reference environment in steps<a class="zola-anchor" href="#start-your-reference-environment-in-steps" aria-label="Anchor link for: start-your-reference-environment-in-steps">🔗</a></h2>
<h3 id="first-level-automate-what-is-easy-and-document-what-is-hard">First level: Automate what is easy and Document what is hard<a class="zola-anchor" href="#first-level-automate-what-is-easy-and-document-what-is-hard" aria-label="Anchor link for: first-level-automate-what-is-easy-and-document-what-is-hard">🔗</a></h3>
<p>You need to have a step-by-step set of commands to run that walk you through downloading and installing your tools in order to enable others to build their own reference environment. Tools and libraries installed through a package manager, and anything manually installed off a website.</p>
<p>The goal of documenting the setup of your local environment can then be further streamlined the more you automate the step-by-step into scripts that you can provide.</p>
<p>The experience you want to cultivate is a relatively short step from a fresh clone, to building and modifying code.</p>
<p>If you use a language that has its own package manager used to compile, such as Javascript’s NPM, Java’s Maven or Rust’s Cargo, then you are able to keep track of the names and version numbers of the libraries you use.</p>
<p>I recommend that you learn how to use these tools to also remove files created during build. Cached files from previous builds are classic causes of “<a rel="noopener nofollow" target="_blank" href="https://www.leadingagile.com/2017/03/works-on-my-machine/">works on my machine</a>” and may hide otherwise obvious issues. Make a habit of trying to reproduce issues starting from a cache-free build by using the same tools for build to also remove the files and directories it creates.</p>
<p>Not all language package managers come with cleanup functionality, so you may need to install plugins. Otherwise you should use a tool like <code>make</code>, and create separate <code>make build</code> and <code>make clean</code> targets.</p>
<h3 id="second-level-shell-scripts">Second level: Shell scripts<a class="zola-anchor" href="#second-level-shell-scripts" aria-label="Anchor link for: second-level-shell-scripts">🔗</a></h3>
<p>Automation should be driven by makefiles or plain shell scripts like <code>bash</code>/<code>zsh</code>/<code>fish</code>/<code>sh</code> or in windows <code>batch</code> or <code>powershell</code>. The rationale is that these are plain commands that you can copy/paste from your terminal, and into the script. You can always call out to another language like <code>python</code>/<code>ruby</code>/<code>perl</code> or other tools.</p>
<p>But the initial point of functionality should be plain shell scripting through an executable shell script or a <code>Makefile</code>.</p>
<p>If you are working from an existing step-by-step document, then you should have a sequence of commands that you can transfer straight into your script. This is a lot fewer commands for your future users (and future you) to deal with.</p>
<h3 id="third-level-configuration-management">Third level: Configuration management<a class="zola-anchor" href="#third-level-configuration-management" aria-label="Anchor link for: third-level-configuration-management">🔗</a></h3>
<p>There are tools that specifically deal with configuration management of your operating system / development environment. One of their main selling points is declaring the desired state through config files instead of scripting everything step-by-step. The tools read the config files and check the system for differences. If differences are found, then these tools take specific actions in order to attempt to bring the reality of the system state to be the same as the config.</p>
<p>The config files lend themselves into achieving automation that is “<a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Idempotence">idempotent</a>”. A simple explanation of idempotency with respect to configuration automation is experienced when running the automation twice in succession. The first time, an automation tool takes a moment to execute because work is being performed. The second time, the tool discovers that there is no work to be done so the execution time is significantly reduced.</p>
<p>Examples:</p>
<ul>
<li>Ansible</li>
<li>Saltstack</li>
<li>Puppet</li>
<li>Chef</li>
</ul>
<p>In your shell scripts, you need to install these config management tools first. But then the tools can replace significant volumes of shell scripts in many cases because their idempotent properties lead to more efficient execution. The more popular tools have an added benefit of wide support of different OSes, allowing your configs to also widely support different OSes.</p>
<h3 id="next-level-portable-reference-environments-via-virtual-machines-containers">Next level: Portable Reference Environments via Virtual Machines & Containers<a class="zola-anchor" href="#next-level-portable-reference-environments-via-virtual-machines-containers" aria-label="Anchor link for: next-level-portable-reference-environments-via-virtual-machines-containers">🔗</a></h3>
<p>Building your reference environment within a virtual machine may enable you to be a little less strict about documenting and automating your setup. After you build the environment, you can save a snapshot, back it up or even distribute it to others. This can be convenient and portable in most cases.</p>
<p>You can also use your scripts or config management to configure virtual machines, or containers. Using tools like <a rel="noopener nofollow" target="_blank" href="https://www.vagrantup.com/">Vagrant</a> or <a rel="noopener nofollow" target="_blank" href="https://www.docker.com/">Docker</a> allow you to store the instructions for building virtual machines (via <a rel="noopener nofollow" target="_blank" href="https://www.vagrantup.com/docs/vagrantfile">Vagrantfile</a>) or containers (via <a rel="noopener nofollow" target="_blank" href="https://docs.docker.com/engine/reference/builder/">Dockerfile</a>) alongside your code. This enables potential contribution without the need of installing tools locally because the codebase includes a portable development environment.</p>
<h4 id="platform-specific-caveats">Platform specific caveats<a class="zola-anchor" href="#platform-specific-caveats" aria-label="Anchor link for: platform-specific-caveats">🔗</a></h4>
<p>Virtualization or containerization is not an available option for every platform. In these cases, you may be limited to config management or documentation for your reference environments.</p>
<p>If your project has the following, you may have limited support for a portable reference environment. </p>
<ul>
<li>Apple products</li>
<li>Platform-specific tools like IDEs and/or compilers</li>
<li>Hardware requirements such as GPU</li>
<li>Very old “Legacy” OSes</li>
</ul>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>The specific tools, and libraries you use in development may seem obvious to you. But as a courtesy to an outsider, or even to yourself in the future, you need to provide requirements needed to build your code.</p>
<p>Documentation requires discipline to keep up-to-date. Scripts, config management or even a portable environment such as a virtual machine or a container are even better because they are easier to verify for correctness.</p>
<p>The less critically someone needs to think in order to get started, the easier it will be to focus on what matters. Reference environments are not one-size-fits all. They may be unique per project, or organization. Do future you or future team members a favor and write these details down. It can save a lot of time and avoid works-on-my-machine.</p>
Rust in 20212020-11-17T00:00:00+00:002020-11-17T00:00:00+00:00https://tjtelan.com/blog/rust-2021/<p>I missed the official call for blog post submissions, I still want to share some ramblings of my hopes for Rust in 2021.</p>
<p>I’ll split this up into two sections: What I hope to see from the community, and what I hope to see for myself.</p>
<h2 id="what-i-want-for-the-rust-community-in-2021">What I want for the Rust community in 2021<a class="zola-anchor" href="#what-i-want-for-the-rust-community-in-2021" aria-label="Anchor link for: what-i-want-for-the-rust-community-in-2021">🔗</a></h2>
<h3 id="i-want-to-see-rust-shed-some-of-its-reputation-for-being-hard-to-learn">I want to see Rust shed some of its reputation for being hard to learn<a class="zola-anchor" href="#i-want-to-see-rust-shed-some-of-its-reputation-for-being-hard-to-learn" aria-label="Anchor link for: i-want-to-see-rust-shed-some-of-its-reputation-for-being-hard-to-learn">🔗</a></h3>
<p>When I was learning Rust, I thought it was harder than necessary. But I would admit that it was due to writing strict idiomatic Rust while I was still figuring out how to be productive.</p>
<p>I think learners should give themselves patience to learn Rust by writing bad or mediocre code first, and focus productivity later.</p>
<p>I am not exactly sure how to create the conditions for this outcome. Maybe more people can publish more Rust that looks messy but “just works”.</p>
<p>I know that Rust in Action is being written and it is for an intermediate-level audience. But perhaps this looks more like encouragement to beginners to ignore the warnings from the compiler. Just write Rust code and form your own opinions.</p>
<h3 id="more-blog-posts-from-developers-and-management-using-rust-at-work">More blog posts from developers and management using Rust at work<a class="zola-anchor" href="#more-blog-posts-from-developers-and-management-using-rust-at-work" aria-label="Anchor link for: more-blog-posts-from-developers-and-management-using-rust-at-work">🔗</a></h3>
<p>For 2021, I don’t even care if it is from one of the many blockchain-related companies using Rust. I want to read more experiences that lay out the decision of why Rust was chosen and how the experience has been.</p>
<p>I am especially interested when expertise is in another language and a comparison can be made to the other language. Emphasis when performing a rewrite.</p>
<ul>
<li>Example: <a rel="noopener nofollow" target="_blank" href="https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine">https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine</a></li>
</ul>
<p>It’s great to hear Rust used in complicated, low-level development, but it would be normalizing to read about more trivial types of applications, just as a way to highlight Rust as being good for general purpose use.</p>
<p>Anecdotally, I feel like Go has been growing a reputation as being suitable as a scripting language kind for a while and Rust could also check many of the same boxes.</p>
<h3 id="more-rust-in-devops-tooling">More Rust in DevOps tooling<a class="zola-anchor" href="#more-rust-in-devops-tooling" aria-label="Anchor link for: more-rust-in-devops-tooling">🔗</a></h3>
<p>In my experience, the cultures surrounding Rust and DevOps are very similar. They both seem to attract cult-like appeal that demand adopters to adjust to thinking in a different way than they may have been taught.</p>
<p>The intended reasons are for a “better future experience” that usually alludes to improved cooperation at a non-technical level.</p>
<p>The implementations of Rust or DevOps shift considerations of future problems earlier in development. Possibly experiencing some of the friction earlier.</p>
<p>Rust’s benefits are often placed in a context of pure development benefit, which may be ok for now given it is primarily Devs that are writing code. But as a user, there are a few tools in the space that I’m aware of that support DevOps culture (e.g. Habitat, Linkerd) and I want to see more of them.</p>
<ul>
<li>Link regarding rise of Rust + DevOps: <a rel="noopener nofollow" target="_blank" href="https://d2iq.com/blog/rust-devops">https://d2iq.com/blog/rust-devops</a> </li>
</ul>
<h3 id="better-rust-interoperability-between-languages">Better Rust Interoperability between languages<a class="zola-anchor" href="#better-rust-interoperability-between-languages" aria-label="Anchor link for: better-rust-interoperability-between-languages">🔗</a></h3>
<p>Rust is entering other ecosystems through libraries via <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html#using-extern-functions-to-call-external-code">FFI</a>. People don’t even have to know or care if they are using Rust. </p>
<p>However, if I’m writing the code myself, it would be nice if FFI were done without needing to be so mindful of C as the common language.</p>
<p>C++ and Java come to mind as currently Rust interop is a complicated dance. Rust to Javascript is improving through the Deno runtime, or through WebAssembly.</p>
<p>My motivation is that I’d like to work in the ecosystems where these languages are most often used (Web, Mobile and Embedded), but I stubbornly want to use Rust as much as possible.</p>
<h3 id="more-shared-experiences-from-people-picking-up-rust-as-a-second-language">More shared experiences from people picking up Rust as a second language.<a class="zola-anchor" href="#more-shared-experiences-from-people-picking-up-rust-as-a-second-language" aria-label="Anchor link for: more-shared-experiences-from-people-picking-up-rust-as-a-second-language">🔗</a></h3>
<p>The borrow checker has a reputation that scares people away.</p>
<p>Especially if they are not from a traditional CS/Engineering background. And we should be trying to bring these folks into the ecosystem.</p>
<p>Acceptance of Rust as another language worth knowing by “average devs” would be helped with more beginners writing or talking about it. It’s my observation that a lot of developers who are self-taught or from bootcamps are mostly in the Javascript/Python/Ruby space for mostly Web facing code. But Rust can be a perfect second language to learn about types and memory management.</p>
<p>I believe content from this crowd would be my favorites to read since I think they’ll give valid opinions to someone who has been using Rust for years (like me) now overlook or accept without second thoughts.</p>
<h3 id="increase-of-copy-paste-templates-to-help-bootstrap-common-types-of-projects">Increase of copy/paste templates to help bootstrap common types of projects<a class="zola-anchor" href="#increase-of-copy-paste-templates-to-help-bootstrap-common-types-of-projects" aria-label="Anchor link for: increase-of-copy-paste-templates-to-help-bootstrap-common-types-of-projects">🔗</a></h3>
<p>I have just become aware of <code>cargo generate</code> because of some of the niche project spaces I’m working on: WebAssembly and embedded.</p>
<p>Today, I keep <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples">my own repo</a> where I offer rough working examples using common servers or patterns.</p>
<h3 id="my-technical-wishlist">My technical wishlist<a class="zola-anchor" href="#my-technical-wishlist" aria-label="Anchor link for: my-technical-wishlist">🔗</a></h3>
<ul>
<li>Rust analyzer support for generated code
<ul>
<li>I want support for <a rel="noopener nofollow" target="_blank" href="https://github.com/hyperium/tonic">Tonic</a>’s generated code</li>
</ul>
</li>
<li>Better tools or patterns for debugging Async/Await
<ul>
<li>The debugger just makes me realize how little I understand about Futures.</li>
</ul>
</li>
<li>Native Rust for building games
<ul>
<li>The end goal being Rust on consoles or in AAA games. I know they are sometimes used through FFI. It’s not a light task to use Rust in this manner.</li>
</ul>
</li>
<li>Native Rust for building mobile apps
<ul>
<li>We can technically write apps for both Android and iOS today, but they aren’t exactly popular because building is complicated</li>
</ul>
</li>
</ul>
<h2 id="where-i-plan-to-be-with-rust-in-2021">Where I plan to be with Rust in 2021<a class="zola-anchor" href="#where-i-plan-to-be-with-rust-in-2021" aria-label="Anchor link for: where-i-plan-to-be-with-rust-in-2021">🔗</a></h2>
<h3 id="i-plan-to-write-more-rust-that-other-people-find-useful">I plan to write more Rust that other people find useful.<a class="zola-anchor" href="#i-plan-to-write-more-rust-that-other-people-find-useful" aria-label="Anchor link for: i-plan-to-write-more-rust-that-other-people-find-useful">🔗</a></h3>
<ul>
<li>I hope to have <a rel="noopener nofollow" target="_blank" href="https://github.com/orbitalci/orbital">OrbitalCI</a> into more of a Beta quality, and usable by others for feedback</li>
<li>Publish more crates that allow people to write Rust to interact with existing APIs</li>
</ul>
<h3 id="i-plan-to-have-serious-projects">I plan to have serious projects<a class="zola-anchor" href="#i-plan-to-have-serious-projects" aria-label="Anchor link for: i-plan-to-have-serious-projects">🔗</a></h3>
<p>A few serious projects in upcoming Rust niches so that I can produce more educational content (and hopefully even a product that others may use!)</p>
<ul>
<li>Embedded</li>
<li>WebAssembly</li>
<li>Mobile</li>
</ul>
<h3 id="i-plan-to-start-making-money-with-rust">I plan to start making money with Rust.<a class="zola-anchor" href="#i-plan-to-start-making-money-with-rust" aria-label="Anchor link for: i-plan-to-start-making-money-with-rust">🔗</a></h3>
<ul>
<li>I’ve started my own business in 2020, and I’m not yet making any money. However, I’ve decided to base a lot of my work using Rust. I am enjoying the experience so far. I intend to sharpen my skills throughout 2021 into paid work.</li>
<li>I hope to make some income through Twitch from livestreaming my coding and other platforms from my writing.</li>
</ul>
<h3 id="how-do-i-intend-to-help-the-rust-project">How do I intend to help the Rust project?<a class="zola-anchor" href="#how-do-i-intend-to-help-the-rust-project" aria-label="Anchor link for: how-do-i-intend-to-help-the-rust-project">🔗</a></h3>
<ul>
<li>I’m using Rust to build software for my business (in the DevOps tooling space).</li>
<li>I will be producing more blog posts, with an angle of inviting Python and Javascript programmers to Rust</li>
<li>I plan on making my first code or docs contribution to the Rust project in 2021.</li>
</ul>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>I see a lot of promise in Rust's future in 2021. Much has become stable in 2020 and all that is needed is some time and encouragement.</p>
<p>I have enough confidence that I plan on using Rust as my main language as well as motivation to get more involved with the community.</p>
Is DevOps Automation a zero-sum game?2020-11-10T00:00:00+00:002020-11-10T00:00:00+00:00https://tjtelan.com/blog/devops-zero-sum/<p><a rel="noopener nofollow" target="_blank" href="https://dev.to/tjtelan/is-devops-automation-a-zero-sum-game-3dg7">Link to the original discussion on Dev.to</a></p>
<hr />
<p>Some of my intention is to lean into the ambiguity of the definition of DevOps.</p>
<p>In general, when automation is created and introduced into an organization for the purposes of "progress" for a wide audience, is this improvement gained at the expense of an individual or a team?</p>
<hr />
<p>Here's a made up example. Automatic alerts in a production environment of a web app for a small US company's online order form.</p>
<p>The primary customer live in US timezones. The assumed culture is that outages are to be avoided. 24/7 uptime.</p>
<p>Before alerting was introduced in this org, outages in production might not be known until traditional work hours when more complex workflows.</p>
<p>After alerting was introduced, the possibility of non-work hour wake up calls from alerts increased significantly. On-call/event-based compensation might balance the fairness to the person responsible, but that's a monetary cost paid from the org to offset a forced work-life cost.</p>
<p>In my opinion, the introduction of automation led to compromise by the individual, with practically no benefit to the business to make no change.</p>
DevOps Helps you be a Better Leader2020-11-06T00:00:00+00:002020-11-07T00:00:00+00:00https://tjtelan.com/blog/devops-helps-you-be-a-better-leader/<p><em>This is the first of a series about helping your organization embrace DevOps.</em></p>
<h2 id="devops-is-misunderstood">DevOps is misunderstood<a class="zola-anchor" href="#devops-is-misunderstood" aria-label="Anchor link for: devops-is-misunderstood">🔗</a></h2>
<p>DevOps is buzzwordy – so hot right now. Like all tech development trends, everyone wants it, but everyone also has their own ideas about the kind of problems it will solve. This is a precarious situation, because a lot of people aren’t wrong. But they are unaware that their version of the truth is so narrow that they end up missing the big picture.</p>
<p>In this post, I’ll discuss DevOps at a high level by addressing: </p>
<ul>
<li>What is (and what isn’t) DevOps</li>
<li>How DevOps differs from the “normal” development process you may already have</li>
<li>How practicing DevOps can reduce time and money required for development and operations</li>
<li>How nurturing a DevOps culture will increase the quality of life for everyone working on your product</li>
</ul>
<p>There are many more topics to discuss, but we must stay focused! This should lay the foundation to allow for constructive conversation about how DevOps works in your organization. </p>
<h3 id="who-should-read-this-post">Who should read this post?<a class="zola-anchor" href="#who-should-read-this-post" aria-label="Anchor link for: who-should-read-this-post">🔗</a></h3>
<p>This post is primarily intended for people who need to know the purpose of DevOps because they’re managing a team or hiring for a DevOps skill set. You also might find it helpful if you’re just wondering what DevOps is all about as a developer, operator, student, or in a related, non-technical role.</p>
<h3 id="conventions-and-definitions">Conventions and Definitions<a class="zola-anchor" href="#conventions-and-definitions" aria-label="Anchor link for: conventions-and-definitions">🔗</a></h3>
<p>Let’s start with some definitions and conventions so we’re on the same page.</p>
<p><strong>Environment</strong> or <strong>space</strong> may be used interchangeably. It refers to the location where code is being executed. A laptop or workstation are examples of environments that we can physically touch. But Kubernetes, or “the cloud” are examples of more abstract environments.</p>
<p><strong>Development environment</strong> will refer to any environment that is primarily used by a single person for the purposes of writing code, such as a laptop or a workstation.</p>
<p><strong>Production environment</strong> will refer to any environment that is shared, regardless of whether external customers are using it. An internally used environment used only for testing purposes is a production environment. Big differences between a development environment and a production environment may be complexity or scale at which components are deployed. As well as the amount of targeted automation to perform changes such as deployment or collection of analytics. </p>
<p><strong>Development</strong> <em>(Dev)</em> is a phase and a team we associate with writing code. Specifically, it refers to code that has not yet been released officially to a wider audience. When a project is in development, there is an implication that it may not yet be stable. The people who do the code writing are called Developers, or Dev.</p>
<p><strong>Operations</strong> <em>(Ops)</em> is a phase and a team we associate with building and maintaining the environment(s) used for running code from Development, as well as infrastructure used to support the running code. The people who run code may also be Developers, but they are also commonly known as Operators, Ops, SysAdmins, SREs, Production Engineers, and increasingly (and incorrectly) as DevOps Engineers.</p>
<p><strong>DevOps</strong> is the art of joining principles from traditional software Development and Operations so both can work better together. DevOps is not a phase of development, nor is it a role, despite the emergence of DevOps engineering roles or teams. It is an organization’s professional culture, and emotional intelligence for working together as a single team. The purpose of DevOps is managing expected behavior of the product in production with respect to constant “change of state” (we’ll talk about that soon).</p>
<h2 id="the-problem-devops-solves">The problem DevOps solves<a class="zola-anchor" href="#the-problem-devops-solves" aria-label="Anchor link for: the-problem-devops-solves">🔗</a></h2>
<p>An industry trope is the antagonistic rivalry about whose role, Dev or Ops, is more important. But it doesn’t need to be a battle. DevOps is a culture or method of interaction between Dev and Ops and a direct result is shortened feedback loops between writing code (development) and running code (operations). </p>
<p>Admittedly, “DevOps” is a more complex concept when compared to Dev or Ops, which muddies an otherwise straightforward conversation because the actual tasks of practicing DevOps are not one-size-fits all.</p>
<p>Software is rarely a single moving piece. The benefit is the flexibility to incrementally change the state of any single piece. The “change of state” refers to updates to code or the supporting infrastructure, and effectively communicating those changes to the people involved. As a system grows in complexity, it takes some effort to make changes without disrupting other pieces.</p>
<p>DevOps, as a practice, is about managing the changes to those moving pieces in an organized and intentional manner. An example of where DevOps builds this bridge is deploying to production. Thin encourages releasing and running code in a way that Devs and Ops can anticipate each other’s changes, at a high rate. These procedures are preferably automated, for speed and consistency.</p>
<h2 id="change-the-job-title-on-your-reqs-devops-is-not-a-role">Change the job title on your reqs: DevOps is not a role!<a class="zola-anchor" href="#change-the-job-title-on-your-reqs-devops-is-not-a-role" aria-label="Anchor link for: change-the-job-title-on-your-reqs-devops-is-not-a-role">🔗</a></h2>
<p>Unfortunately, DevOps is increasingly used as a catch-all for all engineering tasks that do not have an owner. The result is that it starts to lose any specific meaning.</p>
<p>A recent (and regrettable) trend in job postings is to rebrand SysAdmin or Operations roles into “DevOps engineer” roles. This makes it more difficult for companies and job-seekers alike to make a good match.</p>
<p>Case in point: coding is often listed as a requirement for a “DevOps Engineer” role, but if that role is actually an operations role, there will be few opportunities to write code. This means that new hires joining with the expectation that their role includes coding will feel misled, and turnover will be increased.</p>
<h2 id="what-does-a-solution-look-like">What does a solution look like?<a class="zola-anchor" href="#what-does-a-solution-look-like" aria-label="Anchor link for: what-does-a-solution-look-like">🔗</a></h2>
<p>So what is DevOps if not a role? Let’s describe practicing DevOps through the actions of the members of the technical organization.</p>
<h3 id="devops-enabled-dev-vs-pure-development">DevOps-enabled Dev vs Pure Development<a class="zola-anchor" href="#devops-enabled-dev-vs-pure-development" aria-label="Anchor link for: devops-enabled-dev-vs-pure-development">🔗</a></h3>
<p>Unlike pure development, DevOps is primarily about improving processes and analytics capabilities to remove friction from developers’ workflows, outside of writing code. It is not about product functionality or quality of the product code. For example, Devs define the terms of success or failure of the behavior of their code in production. But thanks to DevOps practices, they can analyze that data to inform their next actions. </p>
<p>Devs in a DevOps practice have a shorter onboarding process as well. Common tasks such as software requirements for build code and deploying code are documented through automation that is kept up to date. With this, new devs can focus on the codebase on day one, not spending all day reading documentation.</p>
<h3 id="devops-enabled-ops-vs-pure-operations">DevOps-enabled Ops vs Pure Operations<a class="zola-anchor" href="#devops-enabled-ops-vs-pure-operations" aria-label="Anchor link for: devops-enabled-ops-vs-pure-operations">🔗</a></h3>
<p>Unlike pure operations, DevOps is proactive about observing how running code is behaving from within the code rather than only observing results. The purpose is to maximize the value of human time by only requiring a person to look into a problem if existing automated strategies fail to resolve it.</p>
<p>Additionally, the introduction of Continuous Integration/Continuous Delivery (or Deployment) pipeline (aka CI/CD) is an important collection of automation shared between Devs and Ops. The Devs know how to push code for build for the CI, and the Ops know how to pull build artifacts for deployment. </p>
<h3 id="devops-enabled-management">DevOps-enabled management<a class="zola-anchor" href="#devops-enabled-management" aria-label="Anchor link for: devops-enabled-management">🔗</a></h3>
<p>There is a lot to be said about DevOps-enabled management, but I will keep it brief (but stay tuned, this will be the topic of a future post). For now, I will be focusing on communication, quality of life of your employees, and a short aside on saving money on cloud operational expenditure.</p>
<p>As a product manager, DevOps directly serves the priorities and product roadmap, as they will be naturally driven from the analytics of your production environments. More objective discussion can be had with your developers and operators when you can provide more specific details and requirements to what is motivating your decision making as a manager. Data about feature usage or responsiveness can more easily be measured and placed onto a dashboard if the metrics to be tracked are known during development, rather than after a feature is complete. This production data can help facilitate discussions with your stakeholders.</p>
<p>As a hiring manager, DevOps enables you to be more specific about what it is you are looking for in job applicants. If you’re looking for a candidate with broad skills, you can justify why with actual functions that need served. Otherwise, you can also be much more specific with need for specialized roles since the functions of your direct reports feed into the quality of your analytics. </p>
<p>Lastly, if you operate in cloud infrastructure such as AWS or Azure, your DevOps practice should condition you to analyze your compute usage for ways to save money. It can show you when you may be able to shut down compute resources for the night to save costs on guaranteed idle time that you would otherwise be paying for. Automation can automatically turn them back on for the work day if needed by devs, but otherwise, you may find that these resources are often unintentionally abandoned by devs.</p>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>If you are a manager, how do you start a DevOps practice? You start by consistently being data-driven through requests for data from your employees. Show them how the data influences your decision making. The consistency of your requests ought to naturally lead to automation of data collection.</p>
<p>Remember that DevOps is more than operations. It is a professional workplace culture that starts with management through commitment to improve communication. </p>
<p>To do both drive and support this, management must be very specific about what information they need from their product, then align their processes to reinforce these needs. Realize that this kind of organization of processes is optimal. The perceived constraints your employees experience provides very clear expectations for team success while allowing individual opportunity to be creative and fulfilled without setting expectations that it is any individual person’s responsibility.</p>
<p>DevOps is a practice that produces solutions that scale to the needs of the organization and humanizes the people who create solutions. It allows people to specialize their craft while providing a structure to let them trust that the rest of their surroundings (infrastructure) will be ready to accept any of their new creations.</p>
Bottle rockets in my DevOps? It’s more likely than you think.2020-10-29T00:00:00+00:002020-10-29T00:00:00+00:00https://tjtelan.com/blog/bottle-rockets-in-my-devops/<p>Why is DevOps such a hot trend when many find it hard to concretely define? It is just for big companies? Could you be practicing DevOps as a small software organization and not even know it? What are the outputs of DevOps activities? How do you explain DevOps to your less-technical colleagues?</p>
<p>To help address those questions, I want to tell you a story not about software, but instead about bottle rockets. We’ll take a look at typical DevOps activities and how feedback loops can naturally form and provide opportunities for improvement using DevOps thinking.</p>
<hr />
<p>Imagine you are building a company that makes and shoots off fireworks. You’re just starting out, so you’re the one making the fireworks.</p>
<p>The firework is aimed, a fuse is lit, the rocket launches, and features are displayed to your customers. You intentionally design pyrotechnics with an expected outcome. Your customers love the loop-de-loops, whistling, and colorful sparks!</p>
<p>At first, you’re just building customized fireworks by yourself with cheap kits. Based on positive feedback from customers, you start to develop a specific type of firework. </p>
<p><em>This is essentially how all software products start.</em></p>
<p>You soon outgrow the capabilities and customizability offered through the kits, and to save on money you develop your own process using raw materials. You build and package every single unit in your garage, and have fireworks shows in a field next to your house. People are buying tickets to your fireworks shows, regardless of how the product was built -- they just want a cool explosion (and yours are the coolest). </p>
<p>What your audience doesn't see are the many experiments and failures that went into building their experience. However, since you work alone, any mishaps are in your private space and they are easy to put out with a fire extinguisher and a bucket of water.</p>
<p><em>This is similar to software development.</em></p>
<p>Your company wants to expand their market size and offerings. You hire new people to help you build more of what currently sells, as well as help create new designs based on feedback from customers. Building in your garage no longer suits the needs of the business. </p>
<p><em>This is similar to the start of <a rel="noopener nofollow" target="_blank" href="https://www.geekwire.com/2014/amazon-20-years-garage-startup-global-powerhouse/">some</a> <a rel="noopener nofollow" target="_blank" href="https://news.microsoft.com/announcement/microsoft-is-born/">legends</a> in <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/HP_Garage">software</a> <a rel="noopener nofollow" target="_blank" href="https://mashable.com/2013/09/27/google-garage-anniversary/">development</a>.</em></p>
<p>You move into a bigger warehouse with an indoor launchpad. You teach the new people how you work. You build and package units by hand. The new people are not as good at this as you are and they don’t have your intuition, so the quality of output is not as consistent. </p>
<p>You’re now launching your bottle rockets from inside the warehouse, through a window. The inconsistencies sometimes cause a small fire so everyone has to stop what they’re doing to help put it out.</p>
<p>You spend the money or time and effort to create special tools so that they don’t needlessly make mistakes.</p>
<p><em>This is similar to <strong>dev tooling</strong>. Dev tooling is domain-specific software or automation created for the purpose of helping developers keep their focus on the details of their problem. It also helps developers to offload the mental burden of tedious tasks such as data processing, reformatting, or validation, onto their tools.</em></p>
<p>Your company now creates many different types of fireworks, and has showcase displays every week. Every product uses specialized parts so it can be a scramble to get everything done in time. One of your new bottle rocket designers discovers that with a few small modifications, many of your products could use the same, adjustable fuse. Now, many firework designs can be made at the same time, and each fuse can be adjusted in the field prior to ignition. You apply this strategy to other parts and this improves your manufacturing efficiency.</p>
<p><em>This is similar to <strong>microservice architecture</strong>, a style of service-oriented architecture design that enables an application to be composed of several small modules organized by business capabilities. These modules can be developed and updated ad-hoc without requiring a full redeployment of all components.</em></p>
<p>You get late notice there’s going to be a huge fireworks expo very soon, and your company’s unique fireworks will be the star of the show. </p>
<p>You return to the product lines to assist with packing units. It’s a terrible experience. </p>
<p>The work is manual, and for some of the tasks you realize that the same tools can be used by an assembly machine and performed with more precision and consistency. With some extra effort, more machines could be constructed to create units for quality control testing, and package units for shipping. The process of assembly, packing, and launch preparation is now more standardized, and your human workers can focus on designing better fireworks instead of remembering the exact order and specifications of assembly.</p>
<p><em>This is similar to <strong>CI/CD</strong> (continuous integration / continuous delivery/deployment). CI/CD is a process that connects development (writing the code) to operations (running the code).</em></p>
<p><em>Typically initiated by a developer pushing code, which builds, tests. This process is called Continuous Integration (CI). The end of a CI process that results in an artifact ready to be deployed is called Continuous Delivery. Going one step further to an automatic deployment of the artifact into production is called Continuous Deployment.</em></p>
<p>As your rockets get bigger, you notice that the aiming from the launchpad isn’t going as well as it used to. With the smaller rockets, you were able to eyeball the right trajectory, but this isn’t working as well with these more complex fireworks. Due to your previous standardizations of fuses, the same fuse modules are used for all the rockets. Therefore we can narrow our investigation of the trajectory issues specifically to construction of the complex fireworks. Quality control is now also improved because there are fewer variants introduced. Changes are easier to track and correct using information from internal quality control and from customers.</p>
<p><em>This is <strong>observation through metrics and monitoring</strong>. The trajectory was noticed not only holistically because of regular launches internally, and by customers. But also objectively, because we’ve collected data on the distances experienced in a variety of conditions.</em></p>
<p>You’re building bigger and better bottle rockets, and your customers are thrilled with the exciting displays you’re putting on! They’ve never seen brighter colors or more interesting designs!</p>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>Using this somewhat silly story as a stand-in for the software development and deployment process, you can see that DevOps activities are often part of a developing and growing system. With every deployment, you’re trying to shoot a bottle rocket out of the window. </p>
<p>Picking and choosing DevOps activities to optimize is normal for small but growing software organizations. Be pragmatic. Adopt solutions when you have problems they solve. </p>
<p>So how do you know what you actually need? You’re likely already addressing the needs you have (but in a much less efficient way). Interested in investing in your team’s DevOps practice? Here are some starting points to think about:</p>
<ul>
<li>If your team spends a lot of time moving between projects, consider bulking up your <strong>dev tooling</strong>.</li>
<li>If you frequently find yourself solving the same problem repeatedly or copy/pasting and lightly modifying service code, investigate implementing <strong>microservice architecture</strong> to do that job for you.</li>
<li>If you rely on a single developer’s machine or struggle to consistently build or release minor changes, <strong>CI/CD</strong> could be what you’re missing.</li>
<li>If you make changes based on feelings and not with data, then improve your product observation through collecting <strong>metrics and monitoring</strong>.</li>
</ul>
<p>A cohesive strategy for packaging, deployment, and response is essential as growth occurs. As the rate of updates increase, the rate of outage-causing failures climbs proportionally. </p>
<p>It is fatally optimistic to rely on the hope that your product is absent of critical bugs. Reactivity is not a dependable plan, especially in the high-stress situations caused when your bottle rocket explodes on the launchpad. Instead, assume that things will inevitably go wrong sometime, somewhere. Create tangible strategies that bridge writing code (development) and running code (operations) through common language and goals, and rely on procedures that considered the important details back when stress levels were not high. </p>
My Hacktoberfest 2020 is Complete2020-10-19T00:00:00+00:002020-10-19T00:00:00+00:00https://tjtelan.com/blog/my-hacktoberfest-2020-is-complete/
<div class="blog-image">
<img src="[object]" alt="Hacktoberfest 2020 complete - banner" />
</div>
<p>This is a followup post to my <a href="https://tjtelan.com/blog/first-time-hacktoberfest-2020/">previous Hacktoberfest related post</a>.</p>
<p><strong>TL;DR - Officially I’ve completed Hacktoberfest 2020!</strong></p>
<p>The beginning of the month was a bit rough, and while there were good intentions, the level of communication to participants was not up to par. But eventually the news of the changes rolled out in a much more official way! For what it’s worth, I approve.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Header with update announcing event being opt-in" /><figcaption>Better late than never</figcaption></figure>
</div>
<p>The change to make eligible repos opt-in will hopefully provide less of an unexpected disruption to open source maintainers. I imagine the spike in activity will belong to repos who want the commits. Also I hope the intended effect will cut down on greedy devs gaming the rules with low quality contributions to get free stuff.</p>
<p>The extended review time was still 2 weeks, but a couple days later after <a rel="noopener nofollow" target="_blank" href="https://tjtelan.com/blog/first-time-hacktoberfest-2020/">my post</a>, I got an email acknowledging my 4 PRs. I’ve just been waiting for the clock to run out this entire time.</p>
<p>However, the time is here. Officially I’ve completed Hacktoberfest 2020!</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Congrats message in my profile" /><figcaption>Congrats message in my profile</figcaption></figure>
</div>
<h2 id="my-strategy">My strategy<a class="zola-anchor" href="#my-strategy" aria-label="Anchor link for: my-strategy">🔗</a></h2>
<p>If you have never contributed to open source before, I can share what I usually do. I took a simple approach.</p>
<p>I crawled the Github search for <a rel="noopener nofollow" target="_blank" href="https://github.com/search?l=&p=1&q=label%3Ahacktoberfest+state%3Aopen+no%3Aassignee+is%3Aissue+language%3ARust&ref=advsearch&type=Issues">Hacktoberfest labeled repos that were written in Rust</a> because I was only interested in writing in that language.</p>
<p>There were quite a lot, so I picked a handful of the smallest Rust projects where the repo owner asked for some <strong>very specific</strong> contributions that were clear enough that it didn’t require me to ask questions before getting started. All things considered, it only took me a total of 2-3 hours to write code.</p>
<p>I did a bit of overcommunication with the repo maintainer to be friendly, but also to give the impression that changes can still be made with their feedback. I wanted to offer an opportunity for them to give me feedback so they would be happy with what they would get merged in.</p>
<p>If you didn’t know, you can <a rel="noopener nofollow" target="_blank" href="https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue">reference issues from your commit message</a>, which I did. Additionally I linked the PR to the issue, and I left a message in the issue itself describing my thought process. There were small changes requested by the maintainers, though every PR was eventually accepted and merged.</p>
<h2 id="the-prize">The Prize<a class="zola-anchor" href="#the-prize" aria-label="Anchor link for: the-prize">🔗</a></h2>
<div class="blog-image"><figure>
<img src="[object]" alt="Hacktoberfest Forest 2020 logo" /><figcaption>Am I a digital lorax? Because I wrote code for the trees</figcaption></figure>
</div>
<p>So I’ve decided to plant a tree instead of receiving a shirt. It was nice that there was an alternative.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Order form for claiming plant a tree prize" /><figcaption>There was only one color and one size. Fun!</figcaption></figure>
</div>
<p>I don’t know what’s next. After completing the form I was told to await instructions from Digital Ocean for how to continue the process of planting the tree.</p>
<p>Either way, this is the end of the line. Hacktoberfest 2020 is done for me. I feel happy to have made a positive impact.</p>
<p>Thanks to Digital Ocean, Intel, Dev.to and the Hacktoberfest team for making this year happen!</p>
<hr />
<p>In the future I will likely crawl the Github search for small Rust projects to contribute to live on <a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/tjtelan">my Twitch stream</a>. If that interests you, or if you’d like me to contribute to your repo, then I’d appreciate it if you’d <a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/tjtelan">give me a follow</a>! </p>
What can I do to feel like I belong in the Tech community?2020-10-13T00:00:00+00:002020-10-13T00:00:00+00:00https://tjtelan.com/blog/belonging-in-the-tech-community/<p>When I was a teenager, in the 00’s, the internet gave me net-positive energy and optimism. It was still very novel, and there was a lot of opportunity to discover niche communities that overlapped with my interests. I felt like part of an in-crowd and it deeply influenced my life.</p>
<p>I have continuously been connected online (Metaphorically. Bc I grew up with dial-up). Much of the novelty has long worn off.</p>
<h2 id="it-s-been-an-exhausting-year">It’s been an exhausting year<a class="zola-anchor" href="#it-s-been-an-exhausting-year" aria-label="Anchor link for: it-s-been-an-exhausting-year">🔗</a></h2>
<p>I was laid off early in the year due to the pandemic. Due to the sudden loss of human interaction, I have been revisiting some of that yearning for connection as I transition to starting a business from scratch (more about that some other time). As a necessity, I am dependent on the internet for the majority of my human interaction. Maybe this is me being mentally old, but I feel the current energy of my echochamber to be more net-negative and I’m hoping I can introduce myself to people who build others up. </p>
<p>Niche communities are difficult for me to discover with the same ease as when I was young. That’s not a reason enough for me to stop looking! So that’s fine. I’ll start looking where I’m at. For professional reasons, I <em>need</em> to start in Tech. So I’ll need to be more specific, because Tech is HUGE.</p>
<hr />
<h2 id="searching-for-my-k-nearest-neighbors">Searching for my k-nearest neighbors<a class="zola-anchor" href="#searching-for-my-k-nearest-neighbors" aria-label="Anchor link for: searching-for-my-k-nearest-neighbors">🔗</a></h2>
<p>I’ll spare you the rest of the thought exercise. I’ve been focusing my efforts within the Rust community. Why? It’s reignited my joy in writing code again, and I’m trying to start my business with it. <a rel="noopener nofollow" target="_blank" href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved">And because people love it!</a> </p>
<p>With a few years under my belt, I’ve finally reached a point in my Rust development that I no longer consider myself a beginner and I can be productive. Great! Well, now what?</p>
<h3 id="what-have-i-been-doing">What have I been doing?<a class="zola-anchor" href="#what-have-i-been-doing" aria-label="Anchor link for: what-have-i-been-doing">🔗</a></h3>
<ul>
<li>Regularly posting blog posts with a purpose of educating or sharing an experience as a non-expert</li>
<li>Hanging out around Tech Twitter and occasionally commenting and retweeting</li>
<li>Streaming writing Rust code for a few hours at least once a week</li>
<li>Joining so many Slack/Discord/Matrix/Gitter/IRC communities. Mostly lurking, but participating occasionally</li>
<li>Maintaining a <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/git-url-parse">Rust crate</a> and occasionally asking related projects if they would like to depend on it</li>
</ul>
<h3 id="what-have-i-been-expecting">What have I been expecting?<a class="zola-anchor" href="#what-have-i-been-expecting" aria-label="Anchor link for: what-have-i-been-expecting">🔗</a></h3>
<p>Part of me was hoping eventually after seeing the same same faces, same names… that I would feel like I had joined the community and I’d stop feeling so intimidated. Perhaps that I would have experienced some subtle mutual acknowledgement of acquaintance? (I have since been lowering my expectations, fwiw. I now believe it is my responsibility to initiate the interactions with potential personal meaning or expectation.)</p>
<h3 id="what-have-i-not-been-doing">What have I not been doing?<a class="zola-anchor" href="#what-have-i-not-been-doing" aria-label="Anchor link for: what-have-i-not-been-doing">🔗</a></h3>
<ul>
<li>Actively inviting conversation</li>
<li>Stepping out of my comfort zone to ask for feedback</li>
</ul>
<h2 id="won-t-you-be-my-neighbor">Won’t you be my neighbor?<a class="zola-anchor" href="#won-t-you-be-my-neighbor" aria-label="Anchor link for: won-t-you-be-my-neighbor">🔗</a></h2>
<p>When was the last time you joined a new community? What was the defining moment when you felt like you belonged?</p>
<p>I don’t feel unwelcome in Tech, but I feel invisible. My attempts to extend my reach out to those who might actually care to connect has random-like results. Not for lack of trying. But admittedly, I am placing too much value in establishing consistency before figuring out what works for me.</p>
<p>I don't know what options to try next. I’m not fishing for pity, but I am looking for advice. How do I find my audience? How can my audience find me?</p>
<hr />
<p>Join the discussion on my post at <a rel="noopener nofollow" target="_blank" href="https://dev.to/tjtelan/what-can-i-do-to-feel-like-i-belong-in-the-tech-community-1k7k">Dev.to</a> or on <a rel="noopener nofollow" target="_blank" href="https://twitter.com/ThatTJTelan/status/1316132042476670976">Twitter</a></p>
A first-timer's perspective to Digital Ocean's Hacktoberfest 20202020-10-05T00:00:00+00:002020-10-05T00:00:00+00:00https://tjtelan.com/blog/first-time-hacktoberfest-2020/<h2 id="so-what-is-hacktoberfest">So... what is Hacktoberfest?<a class="zola-anchor" href="#so-what-is-hacktoberfest" aria-label="Anchor link for: so-what-is-hacktoberfest">🔗</a></h2>
<p>Digital Ocean’s <a rel="noopener nofollow" target="_blank" href="https://hacktoberfest.digitalocean.com/">Hacktoberfest</a> is a month-long event over the month of October aimed at generating an increase of contributions to open source projects. Participants are asked to open PRs against public repositories in Github with “positive contributions”. Swag is offered to the first 70k participants to open 4 qualified PRs.</p>
<div class="blog-image"><figure>
<img src="https://media.giphy.com/media/l3vRmVv5P01I5NDAA/giphy.gif" alt="Ed from Cowboy Bebop"/><figcaption>A typical hacker</figcaption></figure>
</div>
<h2 id="and-how-is-it-going">And… how is it going?<a class="zola-anchor" href="#and-how-is-it-going" aria-label="Anchor link for: and-how-is-it-going">🔗</a></h2>
<p>I have mixed opinions that honestly, skew negatively. But I feel like I’m more charitable about my feedback than what I've seen on the internet. I’m not intending to drag Hacktoberfest but I’d like to see improvement.</p>
<p>2020 is my first year participating in Hacktoberfest. I’ve already been contributing more to the open source community over the last few years. Particularly for the Rust programming language. I set out to use this event as an excuse to help the Rust community out.</p>
<h3 id="positives">Positives:<a class="zola-anchor" href="#positives" aria-label="Anchor link for: positives">🔗</a></h3>
<h4 id="the-feeling-of-a-community-effort-to-contribute-to-open-source">The feeling of a community effort to contribute to Open Source<a class="zola-anchor" href="#the-feeling-of-a-community-effort-to-contribute-to-open-source" aria-label="Anchor link for: the-feeling-of-a-community-effort-to-contribute-to-open-source">🔗</a></h4>
<div class="blog-image"><figure>
<img src="https://i.makeagif.com/media/10-05-2020/FTPRH-.gif" alt="From Tom Goes to the Mayor into. Shaking hands. Community Spirit"/><figcaption>A typical open source PR</figcaption></figure>
</div>
<p>Contributions to open source can happen at any time. I think it’s a nice idea to have a time where people who use open source software are encouraged to contribute meaningfully. Especially for anyone who may feel a bit of intimidation to do so. I’m sure a T-shirt and some stickers can feel very validating to someone who was just using open source before, and now they have some evidence that they contributed too.</p>
<h4 id="discovering-smaller-projects">Discovering smaller projects<a class="zola-anchor" href="#discovering-smaller-projects" aria-label="Anchor link for: discovering-smaller-projects">🔗</a></h4>
<div class="blog-image"><figure>
<img src="https://media.giphy.com/media/1FXYMTuKX91hS/giphy.gif" alt="Bob Ross. Beauty is everywhere"/><figcaption>Bob Ross had amazing commit messages</figcaption></figure>
</div>
<p>This was a great opportunity to explore Github for projects that don’t require a lot of experience or commitment in order to help a maintainer out. The repos I offered code to were all small Rust projects that were doing some practical things. It was fun, and I put some effort into what I wrote in addition to having good interactions with the maintainers.</p>
<h3 id="negatives">Negatives:<a class="zola-anchor" href="#negatives" aria-label="Anchor link for: negatives">🔗</a></h3>
<h4 id="this-year-s-hot-fix-to-the-rules-and-the-lack-of-follow-up-communication">This year’s Hot-fix to the rules and the lack of follow-up communication<a class="zola-anchor" href="#this-year-s-hot-fix-to-the-rules-and-the-lack-of-follow-up-communication" aria-label="Anchor link for: this-year-s-hot-fix-to-the-rules-and-the-lack-of-follow-up-communication">🔗</a></h4>
<div class="blog-image"><figure>
<img src="https://media.giphy.com/media/jV4wbvtJxdjnMriYmY/giphy.gif" alt="Spongebob. Communication rainbow"/><figcaption>Talk to your community</figcaption></figure>
</div>
<p>I feel like the communication around the event is not being managed well. Very reactive, and not effective at spreading the word.</p>
<p>A few days after the start of the event, there was a <a rel="noopener nofollow" target="_blank" href="https://github.com/digitalocean/hacktoberfest/pull/596">significant change in the rules</a>. I only happened to learn about it through the grapevine, as opposed to from Digital Ocean or through the Hacktoberfest front page or profile pages. Rules feel like they are inconsistently applied, and it feels unwelcoming.</p>
<hr />
<p>Where this change affected me is their change of minimum review times, and probably the validity of my contributions. But I’m confused about what is supposed to apply to me since I had created all my PRs in the first 3 days.</p>
<p>What makes this more confusing is the conflicting information within the Hacktoberfest FAQ.</p>
<p>For example, this is likely some old information about the review time of PRs. States 7 days.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="From FAQ. Review time of 7 days" /><figcaption>From FAQ. Review time of 7 days</figcaption></figure>
</div>
<p>Later in the same FAQ it says it’s been updated to 14 days. But only those <strong>before October 3, 2020 @ 12:00 UTC</strong></p>
<div class="blog-image"><figure>
<img src="[object]" alt="Also from FAQ. Review time of 14 days" /><figcaption>Also from FAQ. Review time of 14 days</figcaption></figure>
</div>
<p>All 4 PRs were made before this rule change was stated to go into effect (Oct 3 @ 12:00 UTC). The latest being Oct 3 @ 6:26 UTC.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="4 PRs. All earlier than Oct 3 @ 12:00 UTC" /><figcaption>4 PRs. All earlier than Oct 3 @ 12:00 UTC</figcaption></figure>
</div>
<p>Changes to the rules could have been communicated somewhere on the profile page in the legend, but were not. Or through email, like the message I received when I made my first PR.</p>
<p>I just noticed all my maturation times jump from about a week to about 2 weeks. 3 of these were PRs that were already merged too.</p>
<p>And I’m more annoyed and slightly unmotivated to continue in the event.</p>
<p>(But tbh, rather than feel angry, I’m psyched that I got a few PRs merged so quickly!)</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Missed opportunity to communicate that this duration has been updated from 7 to 14 days." /><figcaption>Missed opportunity to communicate that this duration has been updated from 7 to 14 days.</figcaption></figure>
</div>
<h4 id="feeling-like-corporate-sponsored-projects-are-taking-advantage-of-free-labor">Feeling like corporate-sponsored projects are taking advantage of free labor<a class="zola-anchor" href="#feeling-like-corporate-sponsored-projects-are-taking-advantage-of-free-labor" aria-label="Anchor link for: feeling-like-corporate-sponsored-projects-are-taking-advantage-of-free-labor">🔗</a></h4>
<p>I think that the core intention of Hacktoberfest is well meaning. But I'm seeing some non-trivial work in big projects labeled for Hacktoberfest.</p>
<p>I’m not personally driven by the swag of a free T-shirt. I just want to help the Rust community and level up my skills while I'm at it. But I feel weird sitting near loud, <a rel="noopener nofollow" target="_blank" href="https://blog.domenic.me/hacktoberfest/">negative opinions</a> that I partially agree with. (Although my feelings are a lot more charitable and less intense)</p>
<p>Before <a rel="noopener nofollow" target="_blank" href="https://github.com/digitalocean/hacktoberfest/pull/596">the silent rule changes</a>, the premise of projects that must opt-out really placed a huge burden on the maintainers of small projects.</p>
<div class="blog-image"><figure>
<img src="https://media.giphy.com/media/48Osj6XyU0ptQRfh5V/giphy.gif" alt="A typical maintainer during Hacktoberfest"/><figcaption>A typical maintainer during Hacktoberfest</figcaption></figure>
</div>
<p>But I’m mostly in my raw feelings, because I feel like the focused energy of the open source community is being taken advantage of by bigger corporate entities for a T-shirt. They are all on the bandwagon to use free energy from eager and inexperienced developers to contribute to their big name-brand projects even though they have their own funding for development.</p>
<h2 id="but-will-i-participate-next-year">But... will I participate next year?<a class="zola-anchor" href="#but-will-i-participate-next-year" aria-label="Anchor link for: but-will-i-participate-next-year">🔗</a></h2>
<div class="blog-image">
<img src="https://media.giphy.com/media/4TtrENnFsz4EWkU6gz/giphy.gif"/>
</div>
<p>Hard to say at this moment, but what incentive do I have? I can contribute any time to small projects, and I most likely will. I have to thank Digital Ocean for helping me realize that.</p>
<p>The spirit to drive help to small projects is one of my favorite parts of open source, and I think that Hacktoberfest probably started out with that ethos before their audience made it about T-shirts and stickers.</p>
<p>I don’t like the gamified feeling, and maybe that makes this event not for me. The new rule changes will very much change the feel of participation. Probably for the better, since it is opt-in.</p>
<hr />
<p>As I continue to dig into the changes, it appears that there was a <a rel="noopener nofollow" target="_blank" href="https://twitter.com/hacktoberfest/status/1312221208667185153">tweet</a> made prior to the changes, but from my perspective, if you didn’t use Twitter and follow <a rel="noopener nofollow" target="_blank" href="https://twitter.com/hacktoberfest">@hacktoberfest</a>, you’d continue to be lost to the change.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://hacktoberfest.digitalocean.com/hacktoberfest-update">This message</a> appears on their site, but I only got to it through the PR. I don't know how else I would have found it because it isn't on the front page or anywhere I'd easily find.</p>
<p>On one account, I've been told that an email was sent out to highlight rule changes. Pretty late after the fact, but I guess better than never. I cannot confirm this myself, since at the time of this writing, I haven't received any email regarding rule changes.</p>
<hr />
<p>Depending on how the changes affect the mood and the outcome of this year's event, I could probably warm up to the idea of 2021. Either way, I'll still be opening PRs against something.</p>
<p>I'll give credit where it is due to the Hacktoberfest team for responding to abusive participants. BUT I don’t appreciate rules being changed without care to communicate them effectively. I expect better communication from a worldwide event. Please learn from this!</p>
<hr />
<p>Anyway, if you've made it this far, don't wait for October to contribute to open source! </p>
9 steps to rename your default Github branch safely2020-09-30T00:00:00+00:002020-09-30T00:00:00+00:00https://tjtelan.com/blog/github-rename-your-default-branch/<p>Want to migrate your git branches from <code>master</code> to <code>main</code>? Your branch protections, in-progress PRs, and drafts can migrate safely. Follow this simple checklist to confidently make these changes and create a seamless experience for yourself and your developer community.</p>
<h2 id="introduction">Introduction<a class="zola-anchor" href="#introduction" aria-label="Anchor link for: introduction">🔗</a></h2>
<h3 id="what-s-in-a-branch-name">What’s in a (branch) name?<a class="zola-anchor" href="#what-s-in-a-branch-name" aria-label="Anchor link for: what-s-in-a-branch-name">🔗</a></h3>
<p>If you have existing git repos at Github (or any other git hosting platform), you probably have a branch in your repo named <code>master</code>. Starting October 1st, 2020, Github will officially stop their practice of naming the first branch of new repositories <code>master</code>. Instead, the name <code>main</code> will be used from now on. </p>
<p>The usage of <code>master</code> is unfortunately deeply ingrained into those who learned how to use git and developed muscle memory. But did you know that Git (the tool) has no technical requirement that you use <code>master</code> as your default branch name? However, because it is the first branch created when you run <code>git init</code>, it’s often the default used. As a result, hosting platforms such as Github or continuous integration systems like Jenkins or TravisCI create workflows that typically use these defaults as release branches.</p>
<h3 id="can-t-i-just-rename-master-to-main">Can’t I just rename <code>master</code> to <code>main</code>?<a class="zola-anchor" href="#can-t-i-just-rename-master-to-main" aria-label="Anchor link for: can-t-i-just-rename-master-to-main">🔗</a></h3>
<p>Automation and development workflows will make renaming <code>master</code> require more effort than just following <a rel="noopener nofollow" target="_blank" href="https://stackoverflow.com/questions/6591213/how-do-i-rename-a-local-git-branch#6591218">the first Stack Overflow answer you can find</a> to answer the question. </p>
<p>In <a rel="noopener nofollow" target="_blank" href="https://github.com/github/renaming">Github’s official statement</a>, they suggest waiting until later in the year to perform this switch yourself if your repo has any of the following conditions:</p>
<ul>
<li>Open pull requests need to be retargeted to the new branch</li>
<li>Draft releases need to be retargeted to the new branch</li>
<li>Branch protection policies need to be transferred to the new branch</li>
</ul>
<p>Don’t want to wait? While this won’t be as simple as just renaming your branches, you can still perform this switch on your personal or organization’s repos by modifying branch protections, PRs, and draft releases. Follow along to find out how.</p>
<h2 id="migration-strategy">Migration strategy<a class="zola-anchor" href="#migration-strategy" aria-label="Anchor link for: migration-strategy">🔗</a></h2>
<p>I will walk through this migration, step-by-step. Feel free to skip the optional steps if they do not apply to your situation. Note that your new default branch name <em>does not</em> have to be called <code>main</code> – this is just Github’s new default and what I will be using in this example (you can call your new branch <code>steve</code>, for all I care).</p>
<p>This guide is Github-focused, but you should still be able to follow along for other scenarios – these concepts are not exclusive to Github.</p>
<p>Here are the high-level steps we’ll go through:</p>
<ol>
<li>Communicate upcoming changes to collaborators</li>
<li>Mirror/copy <code>master</code> branch to <code>main</code> branch</li>
<li>(Optional) Modify any CI that specifically triggers on <code>master</code> to use <code>main</code></li>
<li>(Optional) Duplicate branch protections from <code>master</code>, and apply them to <code>main</code>.</li>
<li>(Optional) Modify draft releases to target <code>main</code></li>
<li>(Optional) Modify open pull requests to retarget to <code>main</code></li>
<li>Set the default branch to <code>main</code></li>
<li>Delete <code>master</code> branch from local clone</li>
<li>Delete <code>master</code> branch in remote repo</li>
</ol>
<h2 id="1-communicate-upcoming-changes-to-collaborators">1. Communicate upcoming changes to collaborators<a class="zola-anchor" href="#1-communicate-upcoming-changes-to-collaborators" aria-label="Anchor link for: 1-communicate-upcoming-changes-to-collaborators">🔗</a></h2>
<p><strong>This is a very important step!</strong></p>
<p>If your codebase has a significant amount of active development and you plan to rename <code>master</code>, consider the subtle costs and impacts to the workflow. Other developers or operators will need to take action!</p>
<p>The name of a branch has no bearing on how git functions at a technical level. However, the name of a branch <em>can</em> have high importance because of organizational norms of how development and operations are organized. Not communicating changes to your collaborators about renaming your default branch can cause unnecessary confusion when pushing code and loss of trust.</p>
<p>You can’t assume that everyone knows enough about git to make the necessary changes to their workflow to participate. (In which case, you can share this guide!)</p>
<h2 id="2-mirror-copy-master-branch-to-main-branch">2. Mirror/copy <code>master</code> branch to <code>main</code> branch<a class="zola-anchor" href="#2-mirror-copy-master-branch-to-main-branch" aria-label="Anchor link for: 2-mirror-copy-master-branch-to-main-branch">🔗</a></h2>
<p>To start the migration, we want to branch off <code>master</code> into a local branch named <code>main</code> and push <code>main</code> to remote.</p>
<p>We’ll work from a fresh clone (assuming our default branch is <code>master</code>):</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ git clone <url>
</span><span>$ cd <repo>
</span></code></pre>
<p>In the new clone, we will create our new branch, <code>main</code>, and push it to Github.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ git checkout -b main
</span><span>Switched to a new branch </span><span style="color:#d69d85;">'main'
</span><span>
</span><span>$ git push -u origin main
</span><span>Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
</span><span>remote:
</span><span>remote: Create a pull request for </span><span style="color:#d69d85;">'main'</span><span> on GitHub by visiting:
</span><span>remote: https://github.com/tjtelan/example-repo/pull/new/main
</span><span>remote:
</span><span>To github.com:tjtelan/example-repo.git
</span><span> * </span><span style="color:#569cd6;">[</span><span>new branch</span><span style="color:#569cd6;">]</span><span> main -> main
</span><span>Branch </span><span style="color:#d69d85;">'main'</span><span> set up to track remote branch </span><span style="color:#d69d85;">'main'</span><span> from </span><span style="color:#d69d85;">'origin'</span><span>.
</span></code></pre>
<div class="blog-image">
<img src="[object]" />
</div>
<p>With the new branch pushed, we can now start making the platform changes w/ respect to the <code>main</code> branch.</p>
<h2 id="3-modify-any-ci-that-specifically-triggers-on-master-to-use-main">3. Modify any CI that specifically triggers on <code>master</code> to use <code>main</code><a class="zola-anchor" href="#3-modify-any-ci-that-specifically-triggers-on-master-to-use-main" aria-label="Anchor link for: 3-modify-any-ci-that-specifically-triggers-on-master-to-use-main">🔗</a></h2>
<p><em>This step is optional. You can skip it if you don’t have any continuous integration (CI).</em></p>
<p>CI is often triggered on events for specific branches. I’ll go over an example of updating CI workflow using Github Actions.</p>
<h3 id="change-your-trigger-branches">Change your trigger branches<a class="zola-anchor" href="#change-your-trigger-branches" aria-label="Anchor link for: change-your-trigger-branches">🔗</a></h3>
<p>If your CI is triggered on events to your <code>master</code> branch, you’ll obviously need to change that to <code>main</code> for Actions to react to events on the <code>main</code> branch.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span>[</span><span style="color:#b5cea8;">...</span><span>]
</span><span style="color:#569cd6;">on</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">push</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">branches</span><span>: [ </span><span style="background-color:#282828;color:#d69d85;">main</span><span> ]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pull_request</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">branches</span><span>: [ </span><span style="background-color:#282828;color:#d69d85;">main</span><span> ]
</span><span>[</span><span style="color:#b5cea8;">...</span><span>]
</span></code></pre>
<p>Or within your jobs:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span>[</span><span style="color:#b5cea8;">...</span><span>]
</span><span style="background-color:#282828;color:#569cd6;">jobs</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">build</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref == 'refs/heads/main'</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span>[</span><span style="color:#b5cea8;">...</span><span>]
</span></code></pre>
<h3 id="update-any-plugins-referencing-master">Update any plugins referencing <code>master</code><a class="zola-anchor" href="#update-any-plugins-referencing-master" aria-label="Anchor link for: update-any-plugins-referencing-master">🔗</a></h3>
<p>For example, if you happen to be using any of Github’s official plugins, such as <a rel="noopener nofollow" target="_blank" href="https://github.com/actions/checkout">actions/checkout</a>, </p>
<p>Change your Github Actions to act on <code>main</code> instead of <code>master</code></p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span></code></pre>
<p>(or use a tag instead of a branch)</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@v2</span><span>
</span></code></pre>
<p>If you’re using other plugins, you will need to check the repo of those plugins to verify if the author has migrated their <code>master</code> branch to <code>main</code>.</p>
<h2 id="4-duplicate-branch-protections-from-master-and-apply-them-to-main">4. Duplicate branch protections from <code>master</code>, and apply them to <code>main</code><a class="zola-anchor" href="#4-duplicate-branch-protections-from-master-and-apply-them-to-main" aria-label="Anchor link for: 4-duplicate-branch-protections-from-master-and-apply-them-to-main">🔗</a></h2>
<p><em>This step is optional if you don’t have any branch protection rules.</em></p>
<p><code>https://github.com/<account>/<repo>/settings/branches</code></p>
<p>You’ll need to create a new branch protection rule:</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>You’ll probably need to have another window open with the <code>master</code> branch rules open so it’s easier to copy over… At least I did.</p>
<div class="blog-image">
<img src="[object]" />
</div>
<div class="blog-image"><figure>
<img src="[object]" alt="Don't delete the master branch protection rules before you're ready to delete the branch altogeher" /><figcaption>Don't delete the master branch protection rules before you're ready to delete the branch altogeher</figcaption></figure>
</div>
<p>After you’re done with all the branch migration, you can delete the branch protection rules for the <code>master</code> branch. There’s no urgency to do it immediately, but you will need to do this before completing the transition to <code>main</code> (Github won’t let you delete a branch that still has active protection rules.)</p>
<h2 id="5-modify-draft-releases-to-target-main">5. Modify draft releases to target <code>main</code><a class="zola-anchor" href="#5-modify-draft-releases-to-target-main" aria-label="Anchor link for: 5-modify-draft-releases-to-target-main">🔗</a></h2>
<p><em>This step is optional if you don’t have any draft releases currently targeting <code>master</code>.</em></p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>In your repo Releases, select Edit on the draft:</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>In the target dropdown, change the branch to <code>main</code> -- and then Save draft. That’s all there is to it!</p>
<div class="blog-image">
<img src="[object]" />
</div>
<h2 id="6-modify-open-pull-requests-to-retarget-to-main">6. Modify open pull requests to retarget to <code>main</code><a class="zola-anchor" href="#6-modify-open-pull-requests-to-retarget-to-main" aria-label="Anchor link for: 6-modify-open-pull-requests-to-retarget-to-main">🔗</a></h2>
<p><em>This step is optional. You can skip this step if you don’t have existing PRs targeting <code>master</code>.</em></p>
<p>So you have existing PRs against <code>master</code>?</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>This is an easy fix! You can just edit the PR to use your new branch as the base branch.</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>And retarget the base branch to <code>main</code></p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>And then confirm changing the base branch.</p>
<div class="blog-image">
<img src="[object]" />
</div>
<h2 id="7-set-the-default-branch-to-main">7. Set the default branch to <code>main</code><a class="zola-anchor" href="#7-set-the-default-branch-to-main" aria-label="Anchor link for: 7-set-the-default-branch-to-main">🔗</a></h2>
<p>Once you’ve got the <code>main</code> branch pushed up to Github, it is easy to set <code>main</code> to be the new default branch.</p>
<p><code>https://github.com/<account>/<repo>/settings/branches</code></p>
<p>In your repository settings, click the drop-down menu, and select <code>main</code>.</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>Click Update.</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>Click <code>I understand, update the default branch</code></p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>If your organization needs more time to migrate away from <code>master</code>, you can pause after this step. Commits will need to flow into both <code>main</code> and <code>master</code> until it has been fully transitioned. This may be briefly inconvenient, but it will allow processes to continue until <code>master</code> can be deprecated.</p>
<h2 id="8-delete-master-branch-in-remote-repo">8. Delete <code>master</code> branch in remote repo<a class="zola-anchor" href="#8-delete-master-branch-in-remote-repo" aria-label="Anchor link for: 8-delete-master-branch-in-remote-repo">🔗</a></h2>
<p>You have two options to delete the master branch.</p>
<h3 id="through-the-web-browser">Through the web browser<a class="zola-anchor" href="#through-the-web-browser" aria-label="Anchor link for: through-the-web-browser">🔗</a></h3>
<p>On the code tab, click the branch selection drop-down and click the <code>View all branches</code> link</p>
<div class="blog-image">
<img src="[object]" />
</div>
<p>Note: If you had branch protection rules for <code>master</code>, you will need to delete them prior to this step. Otherwise Github will prevent the branch from being deleted.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="The master branch cannot be deleted because of its branch protection rules" /><figcaption>The master branch cannot be deleted because of its branch protection rules</figcaption></figure>
</div>
<p>At the branch listing, find one of the rows with <code>master</code> listed, and click the trash icon on the right.</p>
<p><strong>Warning!</strong> There is no additional prompt after clicking this icon. But you have the option to restore the branch while you are still on the page. However, if you’ve made it this far, deleting it is probably what you want</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Deleting the master branch protection rules" /><figcaption>Deleting the master branch protection rules</figcaption></figure>
</div>
<div class="blog-image"><figure>
<img src="[object]" alt="The master branch can now be deleted" /><figcaption>The master branch can now be deleted</figcaption></figure>
</div>
<h3 id="through-the-cli">Through the cli<a class="zola-anchor" href="#through-the-cli" aria-label="Anchor link for: through-the-cli">🔗</a></h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ git push --delete -u origin master
</span><span>To github.com:tjtelan/example-repo.git
</span><span> - </span><span style="color:#569cd6;">[</span><span>deleted</span><span style="color:#569cd6;">]</span><span> master
</span></code></pre>
<h2 id="9-delete-master-branch-from-local-clone">9. Delete <code>master</code> branch from local clone<a class="zola-anchor" href="#9-delete-master-branch-from-local-clone" aria-label="Anchor link for: 9-delete-master-branch-from-local-clone">🔗</a></h2>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ git branch -D master
</span></code></pre>
<p><img src="https://media.giphy.com/media/26u4lOMA8JKSnL9Uk/giphy.gif" alt="Spongebob" /></p>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<h3 id="a-branch-by-any-other-name-would-still-merge-as-clean">A branch by any other name would still merge as clean<a class="zola-anchor" href="#a-branch-by-any-other-name-would-still-merge-as-clean" aria-label="Anchor link for: a-branch-by-any-other-name-would-still-merge-as-clean">🔗</a></h3>
<p>The names of the branches don’t have any bearing on the functionality of git. But the practice of using offensive terminology to describe defaults or importance needs to end. Plus, <code>main</code> as an initial branch name is a better descriptive label for how the branch is conventionally used.</p>
<p>If you and your collaborators are able to cease the usage of <code>master</code> as your default branch without needing to rely on Github, your git host provider, or your operations team, I encourage you to do it.</p>
<p>The git project <a rel="noopener nofollow" target="_blank" href="https://sfconservancy.org/news/2020/jun/23/gitbranchname/">will soon stop using <code>master</code></a> as the first branch created by <code>git init</code>. So your organization should prioritize this migration as well as making the necessary changes in your tooling and automation sooner rather than later.</p>
Reflections after 2 months of live coding2020-09-22T00:00:00+00:002020-09-22T00:00:00+00:00https://tjtelan.com/blog/reflections-after-2-months-of-live-coding/<p>I started streaming on Twitch this summer. I have been streaming Rust development once a week.</p>
<p>The opportunity presented itself for reasons related to the current state of travel industries (and the economic consequences that followed). It has been an interesting learning experience. </p>
<p>After experimenting with something, it’s always good to take time to stop and reflect on what has been successful and where there’s room for improvement. The time I’ve spent so far has been challenging and confusing – but also enjoyable. This post is for anyone interested in the experiences of a beginning streamer, or anyone considering dipping their toes into the Twitch pool.</p>
<p>I’ll organize my reflections into 3 sections:</p>
<ul>
<li>Things completely in my control</li>
<li>Short-term objectives that I can influence</li>
<li>Long-term aspirations</li>
</ul>
<p><em>If you’re interested how I got started, <a href="https://tjtelan.com/blog/what-i-learned-running-a-live-programming-stream-from-linux/">I wrote a post about it that you can check out!</a></em></p>
<h2 id="things-completely-in-my-control">Things completely in my control<a class="zola-anchor" href="#things-completely-in-my-control" aria-label="Anchor link for: things-completely-in-my-control">🔗</a></h2>
<p>Like most things in life, there’s not much completely within my control. But I can tinker with my channel's look and feel, the time and duration I stream, and what I work on during each stream.</p>
<h3 id="content">Content<a class="zola-anchor" href="#content" aria-label="Anchor link for: content">🔗</a></h3>
<p><strong>Working on a project provides natural boundaries.</strong> Before beginning a stream, I don’t need to know 100% what I’m going to accomplish or how. But I’m more mentally present if I am not thinking too much about what I need to do next. To help with that, I like to plan a predetermined, single direction per stream. I choose a reasonably-sized goal to work on for each stream, and try to roughly break down the steps so I have a small map to follow during the stream.</p>
<p>I also expose project management over stream (via Github projects), which keeps me focused on individual tasks and keeps my stream content on track.</p>
<p><strong>Learn from my mistake.</strong> If you cross-stream to YouTube, be mindful of Content ID. I experimented with background music and naively loaded up another YouTube steam for a short moment. Don’t do this! Pretty quickly it was automatically flagged. Look for royalty-free music instead.</p>
<h3 id="viewer-experience">Viewer experience<a class="zola-anchor" href="#viewer-experience" aria-label="Anchor link for: viewer-experience">🔗</a></h3>
<p>As I’ve continued on, I've put more effort into improving the experience of viewing my stream These are my key takeaways:</p>
<p><strong>Know your limits.</strong> I’ve learned that 3 hours is about my maximum session length. Any longer and I start to feel low energy and my voice gets tired from all the talking. Not to mention, my current viewership and audience engagement doesn't justify being online for that long.</p>
<p><strong>Show your process</strong>. There are two steps (where I’m not actively coding) that I include on my stream as part of my process: diagrams and project management. </p>
<ul>
<li>Writing code is a lot of effort, especially if you don’t know where you’re going. For me, drawing out diagrams of structures or algorithms is like writing an outline before I start coding. I’ve been learning Figma to make nice diagrams for streams.</li>
<li>If I’m practicing project management, I’ll do it live. Lately, I’ve been using Github projects to manage my roadmap and planning.</li>
</ul>
<p>This is an opportunity to show real parts of my process, and there’s no reason to hide that from viewers. </p>
<p><strong>If you don’t know, now you know.</strong> If I don’t know something, I'll open a browser and look up documentation and examples or ask my viewers. It won’t benefit anyone if I’m just struggling. Plus, sometimes the audience can help out . Looking for answers is also a very normal part of coding – no one knows everything, and searching for things effectively is a skill in and of itself.</p>
<p><strong>Talk it out.</strong> It takes some practice, but speaking out my thought process has definitely from when I first started. I realized as I was watching other live-coding streamers for research that it’s not exactly interesting to watch in silence as someone types. People usually will help if they know what you're trying to figure out, or ask questions to feel involved.</p>
<p><strong>Don’t stress.</strong> I also found it helpful to turn off the viewers count if you have the stream manager open. It serves no real time purpose and causes unnecessary mental stress if you're alone in your stream. Always talk like someone is watching.</p>
<p><strong>Back in five.</strong> Have OBS scenes to switch to for bathroom breaks or intentional moments away from the keyboard. I think this is just polite as a non-gamer stream. It’s a bit of a bummer to open up a stream to an empty chair and no indication of when the person will be coming back.</p>
<h2 id="short-term-objectives-i-can-influence">Short-term objectives I can influence<a class="zola-anchor" href="#short-term-objectives-i-can-influence" aria-label="Anchor link for: short-term-objectives-i-can-influence">🔗</a></h2>
<p>I have set a few new long-term goals for myself. But I'll always need to deal with the same obstacles to broadcasting effectively, and get viewers to agree to hang out and talk to me.</p>
<h3 id="asking-myself-questions-first">Asking myself questions first<a class="zola-anchor" href="#asking-myself-questions-first" aria-label="Anchor link for: asking-myself-questions-first">🔗</a></h3>
<p><strong>Practice makes perfect.</strong> If you are going to be exploring a new library on your stream, then practice using it for a non-trivial amount of time prior to broadcast. This will serve the purpose of defining a direction for the stream, in addition to building familiarity of where some of the pitfalls are in case of roadblocks. Having some familiarity with the tools or systems I’m working with also makes me feel more confident in demonstrating it, even when I don’t know how to do everything I plan on doing.</p>
<p><strong><human> has entered the chat.</strong> People who chat will often not talk about the code you're writing. This isn't a negative! People usually just want to get to know me. Keep in mind, Twitch is not a lecture hall (unless that’s actually what your stream actually is).</p>
<p>Yes, it can feel a little disruptive to your train of thought if you’re only prepared to talk about what you’re working on. But that's another reason why to prepare with a plan and practice with any new tools.</p>
<h3 id="reaching-an-audience">Reaching an audience<a class="zola-anchor" href="#reaching-an-audience" aria-label="Anchor link for: reaching-an-audience">🔗</a></h3>
<p><strong>On Twitch.</strong> Twitch does little to promote for you as a small streamer. So experimenting with self-promotion is a must.</p>
<p>The lack of promotion can be a double-edged sword, compared to YouTube. The requirements for Twitch Affiliate are more modest but still take effort. </p>
<p><strong>On YouTube.</strong> YouTube technically promotes you through search results, but I still need to experiment with self-promotion. The bar for reaching YouTube Partner is significantly higher. It definitely requires a different strategy to build an audience than Twitch. I am not actively focused on building my audience here... I just use Restream.io to simultaneously go live on YouTube and Twitch.</p>
<h2 id="long-term-aspirations">Long-term aspirations<a class="zola-anchor" href="#long-term-aspirations" aria-label="Anchor link for: long-term-aspirations">🔗</a></h2>
<p>Truthfully, I don’t have any desire to make a living as a full-time streamer. However, I would like to have diverse sources of income, including anything I can make through streaming and anything I create while streaming. So for that reason alone, it makes it feel more important to take this a little seriously so that I will be able to meet the requirements for Twitch Affiliate.</p>
<h3 id="twitch-specific-channel-improvements">Twitch-specific channel improvements<a class="zola-anchor" href="#twitch-specific-channel-improvements" aria-label="Anchor link for: twitch-specific-channel-improvements">🔗</a></h3>
<p>If I look at the more successful live-coding streamers, they have been on Twitch for a while, they’ve put some effort in their visual branding, and they are on a Twitch team. I believe that these are all realistically achievable for me if I put in the work. Here are improvements I am actively planning or working on:</p>
<ul>
<li>Visual improvements to my OBS scenes</li>
<li>Making my Twitch profile more interesting</li>
<li>Coming up with project ideas that are interesting and/or novel</li>
<li>Determining what time of day to start my stream, so people are more likely to stop by and stick around for a while</li>
<li>Reaching the requirements for Twitch Affiliate</li>
<li>Joining a team: my goal is to join the <a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/team/rustaceans">Rustaceans</a> Twitch team once I reach Affiliate status.</li>
</ul>
<h3 id="networking">Networking<a class="zola-anchor" href="#networking" aria-label="Anchor link for: networking">🔗</a></h3>
<p>I stated before that Twitch doesn’t do much to promote small streamers. Promotion seems to be geared toward creators who are already reaching an audience. Networking is a creative skill I get to build that wouldn’t normally present itself in the Software industry in this capacity. It will certainly benefit me outside the context of streaming. Here are a few paths for networking:</p>
<p><strong>Pre-stream promotion.</strong> It can be hard to find the appropriate venue for pre-stream promotion. I’ve personally found that Twitter is not a great place to promote if you don’t have an existing audience. My engagement is very low on my tweets for good reason. But it isn’t the only option I have. </p>
<p><strong>Find an audience by being active in communities.</strong> I wasn’t really watching other live-coding streams on Twitch prior to doing it myself, but I have noticed that I learn a lot by watching other streamers that are about the same size as myself or larger and taking note of what they do “right”. As well as interacting with them and their audience.</p>
<p>One nice surprise was finding that there are a lot of small channel live-coders or Rust language enthusiasts like me who show up in my chat to talk, and I’ve been able to make some connections this way.</p>
<p>I’m also working on being a more active member of the Rust community. I’ve been using this language for a while. I don’t consider myself an expert, but I do feel like the community can benefit from my voice and experience.</p>
<p><strong>Talk to the people around me.</strong> I hear that Seattle has a pretty active tech scene (For reasons, I need to state that this is a joke).</p>
<h3 id="leveling-up">Leveling up<a class="zola-anchor" href="#leveling-up" aria-label="Anchor link for: leveling-up">🔗</a></h3>
<p>This is a bit of my insecurity presenting itself. But I notice improvements the longer I continue, which encourages me to keep going. I hope to:</p>
<ul>
<li>Improve my skill as someone who can design, plan, and write code. Especially in the domains I spend the most time in: Rust, Python, and Javascript.</li>
<li>Come to terms with my self-consciousness about not being “a good programmer”.</li>
<li>Write more blog posts(I’m near the end of this one. It gets easier the more I do it.)</li>
<li>Producing videos for YouTube as companions to blog posts</li>
</ul>
<hr />
<p>If you’ve made it this far, thanks for reading!</p>
<p>I could not have predicted where I’d be this year. It has not shaped up to be anywhere near what I thought it might be. But I am fortunate that I was able to try something I would have normally not done before.</p>
<p>If you’re interested in how my story continues to procedurally generate, follow me on Twitch.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/tjtelan">https://www.twitch.tv/tjtelan</a></p>
Let’s build a single binary gRPC server-client with Rust in 20202020-08-19T00:00:00+00:002020-10-19T00:00:00+00:00https://tjtelan.com/blog/lets-build-a-single-binary-grpc-server-client-with-rust-in-2020/
<div class="blog-image">
<img src="[object]" alt="The Rust logo plus the gRPC logo" />
</div>
<p>There are plenty of resources for the basics of Rust and for protocol buffers + gRPC, so I don’t want to waste your time with heavy introductions. I want to bring you to action as soon as possible.</p>
<p>If you’re here I’ll make a few assumptions about you.</p>
<ul>
<li>You can write code in another language, but you have an interest in Rust</li>
<li>You have basic familiarity with the command line for simple tasks (like listing files with <code>ls</code>)</li>
<li>You used web service APIs like REST, GraphQL or gRPC in code you’ve written</li>
<li>You’ve <em>skimmed</em> through the <a rel="noopener nofollow" target="_blank" href="https://developers.google.com/protocol-buffers/docs/proto3">official protocol buffers (v3) docs</a> at least once</li>
<li>You are looking for some example code that you can copy/paste and modify</li>
</ul>
<h3 id="goals-for-the-post">Goals for the post<a class="zola-anchor" href="#goals-for-the-post" aria-label="Anchor link for: goals-for-the-post">🔗</a></h3>
<p>My goal is to walk through writing a small async Rust CLI application. It will take user input from a client, send it to a remote gRPC server, and return output to the client.</p>
<p>The finished code is available in my <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples">rust-examples repo</a>, as <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/tree/master/cli-grpc-tonic-blocking">cli-grpc-tonic-blocking</a>. But I encourage you to follow along, as I will narrate changes while I make them.</p>
<h3 id="what-are-we-writing">What are we writing?<a class="zola-anchor" href="#what-are-we-writing" aria-label="Anchor link for: what-are-we-writing">🔗</a></h3>
<p>In this example, I will be writing a remote command-line server/client.</p>
<p>The client will take in a command line command and send it to the server who will execute the command and send back the contents of standard out.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Block diagram with our actors User, Client and Server. Data flows from user to client, then server before looping back." /><figcaption>Diagram of the interaction we'll be working with</figcaption></figure>
</div>
<p>For simplicity sake, this example will wait for the execution to complete on the server side before returning output. In a future post I will demonstrate how to stream output back to a client.</p>
<p>I will show you how to:</p>
<ol>
<li>Parse command line user input</li>
<li>Write protocol buffer message types and service interfaces</li>
<li>Compile protocol buffers into Rust code</li>
<li>Implement a gRPC client</li>
<li>Implement a gRPC server (non-streaming)</li>
<li>Use basic async/await patterns</li>
</ol>
<h4 id="bigger-picture-goals">Bigger picture goals<a class="zola-anchor" href="#bigger-picture-goals" aria-label="Anchor link for: bigger-picture-goals">🔗</a></h4>
<p>This is not just a simple Hello World.</p>
<p>I want to provide an example with a realistic application as a foundation. It has potential to be used for something useful, but keep in mind, this example is just a basic script runner and is not secure. </p>
<div class="blog-image"><figure>
<img src="[object]" alt="A more complex diagram to illustrate how the user, client, server interaction scales. One user, one client, many servers." /><figcaption>This configuration is possible but out of scope</figcaption></figure>
</div>
<p>One could run multiple instances of this server on multiple hosts and use the client to run shell commands on each of them similar to continuous integration tools like jenkins, puppet, or ansible. (Hot take: CI is just fancy shell scripting anyway)</p>
<p>I do not recommend running this code as-is in any important environment. For demonstrative and educational purposes only!</p>
<h2 id="writing-the-command-line-interface">Writing the command line interface<a class="zola-anchor" href="#writing-the-command-line-interface" aria-label="Anchor link for: writing-the-command-line-interface">🔗</a></h2>
<div class="blog-image">
<img src="[object]" alt="The Bourne again shell (BASH) logo" />
</div>
<p>The command line interface is the foundation that will allow us to package our gRPC server and client into the same binary. We’re going to start our new crate with the CLI first.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo new cli-grpc-tonic-blocking
</span><span> Created binary (application) `cli-grpc-tonic-blocking` package
</span><span>$ cd cli-grpc-tonic-blocking
</span></code></pre>
<p>We will use a crate called <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/structopt">StructOpt</a>. StructOpt utilizes the <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/clap">Clap</a> crate which is a powerful command line parser. But Clap can be a little complicated to use, so StructOpt additionally provides a lot of convenient functionality Rust a <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/reference/attributes/derive.html">#[derive] attribute</a> so we don’t have to write as much code.</p>
<p><strong>cargo.toml</strong></p>
<pre data-lang="toml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[</span><span style="color:#808080;">package</span><span>]
</span><span style="color:#569cd6;">name </span><span>= </span><span style="color:#d69d85;">"cli-grpc-tonic-blocking"
</span><span style="color:#569cd6;">version </span><span>= </span><span style="color:#d69d85;">"0.1.0"
</span><span style="color:#569cd6;">authors </span><span>= [</span><span style="color:#d69d85;">"T.J. Telan <t.telan@gmail.com>"</span><span>]
</span><span style="color:#569cd6;">edition </span><span>= </span><span style="color:#d69d85;">"2018
</span><span>
</span><span>[</span><span style="color:#808080;">dependencies</span><span>]
</span><span style="color:#608b4e;"># CLI
</span><span style="color:#569cd6;">structopt </span><span>= </span><span style="color:#d69d85;">"0.3"
</span></code></pre>
<p>In order to bundle our client and server together, we will want to use our CLI to switch between running as a client or running as a server.</p>
<h3 id="some-ui-design-for-the-cli">Some UI design for the CLI<a class="zola-anchor" href="#some-ui-design-for-the-cli" aria-label="Anchor link for: some-ui-design-for-the-cli">🔗</a></h3>
<p>Note: While we are in development you can use <code>cargo run --</code> to run our cli binary, and any arguments after the <code>--</code> is passed as arguments to our binary</p>
<h4 id="starting-the-server">Starting the server<a class="zola-anchor" href="#starting-the-server" aria-label="Anchor link for: starting-the-server">🔗</a></h4>
<p>When we start our server, we want to pass in the subcommand <code>server</code></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo run</span><span style="color:#569cd6;"> --</span><span> server
</span></code></pre>
<h5 id="optional-arguments-for-the-server">Optional arguments for the server<a class="zola-anchor" href="#optional-arguments-for-the-server" aria-label="Anchor link for: optional-arguments-for-the-server">🔗</a></h5>
<p>Most of the time our server will listen to a default address and port, but we want to give the user the option to pick something different.</p>
<p>We will provide the option for the server listening address in a flag <code>--server-addr-listen</code></p>
<h4 id="using-the-client">Using the client<a class="zola-anchor" href="#using-the-client" aria-label="Anchor link for: using-the-client">🔗</a></h4>
<p>When the user runs a command from our client, we want to use the subcommand <code>run</code>. </p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo run</span><span style="color:#569cd6;"> --</span><span> run
</span></code></pre>
<h5 id="required-positional-arguments-for-the-client">Required positional arguments for the client<a class="zola-anchor" href="#required-positional-arguments-for-the-client" aria-label="Anchor link for: required-positional-arguments-for-the-client">🔗</a></h5>
<p>Anything after the <code>subcommand run</code> will be the command we pass to the server to execute. A command has an executable name and optionally also arguments.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo run</span><span style="color:#569cd6;"> -- </span><span><executable> </span><span style="color:#569cd6;">[</span><span>args</span><span style="color:#569cd6;">]
</span></code></pre>
<p>Or to illustrate with how one would use this command w/o cargo if it were named <code>remotecli</code>:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ remotecli run <executable> </span><span style="color:#569cd6;">[</span><span>args</span><span style="color:#569cd6;">]
</span></code></pre>
<h5 id="optional-arguments-for-the-client">Optional arguments for the client<a class="zola-anchor" href="#optional-arguments-for-the-client" aria-label="Anchor link for: optional-arguments-for-the-client">🔗</a></h5>
<p>Just like how our server will have a default listening address and port, our client will assume to connect to the default address. We just want to offer the user the option to connect to a different server.</p>
<p>We will provide the option for the server address in a flag <code>--server-addr</code></p>
<h3 id="the-cli-code-so-far">The CLI code so far<a class="zola-anchor" href="#the-cli-code-so-far" aria-label="Anchor link for: the-cli-code-so-far">🔗</a></h3>
<p>I’m going to break down the current <code>main.rs</code> into their structs, enums and functions to describe how StructOpt is utilized.</p>
<p><strong>Skip down to the next section <a href="https://tjtelan.com/blog/lets-build-a-single-binary-grpc-server-client-with-rust-in-2020/#all-together">All together</a> if you want to review this file in a single code block.</strong></p>
<h4 id="in-parts">In parts<a class="zola-anchor" href="#in-parts" aria-label="Anchor link for: in-parts">🔗</a></h4>
<h5 id="applicationarguments">ApplicationArguments<a class="zola-anchor" href="#applicationarguments" aria-label="Anchor link for: applicationarguments">🔗</a></h5>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// This is the main arguments structure that we'll parse from
</span><span>#[derive(StructOpt, Debug)]
</span><span>#[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"remotecli"</span><span>)]
</span><span style="color:#569cd6;">struct </span><span>ApplicationArguments {
</span><span> #[structopt(flatten)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>subcommand: SubCommand,
</span><span>}
</span></code></pre>
<ul>
<li>
<p>Like the comment says, this will be the main struct that you work with to parse args from the user input. </p>
</li>
<li>
<p>We use <code>derive(StructOpt)</code> on this struct to let the compiler know to generate the command line parser.</p>
</li>
<li>
<p>The <code>structopt(name)</code> attribute is reflected in the generated CLI help. Rust will use this name instead of the name of the crate, which again is <code>cli-grpc-tonic-blocking</code>. It is purely cosmetic.</p>
</li>
<li>
<p>The <code>structopt(flatten)</code> attribute is used on the <code>ApplicationArguments</code> struct field. The result effectively replaces this field with the contents of the <code>SubCommand</code> type, which we’ll get to next. </p>
</li>
</ul>
<p>If we didn’t use flatten, then the user would need to use the CLI like this:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">## No subcommand flattening
</span><span>
</span><span>$ remotecli subcommand <subcommand> …
</span></code></pre>
<p>But with the flattening we get a simplified form without the <code>subcommand</code> literal.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">## With subcommand flattening
</span><span>
</span><span>$ remotecli <subcommand> ...
</span></code></pre>
<p>The reason for this pattern is to allow grouping of the subcommands into a type that we can pattern match on, which is nice for the developer. But at the same time we keep the CLI hierarchy minimal for the user.</p>
<h5 id="subcommand">SubCommand<a class="zola-anchor" href="#subcommand" aria-label="Anchor link for: subcommand">🔗</a></h5>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// These are the only valid values for our subcommands
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub enum </span><span>SubCommand {
</span><span> </span><span style="color:#608b4e;">/// Start the remote command gRPC server
</span><span> #[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>)]
</span><span> StartServer(ServerOptions),
</span><span> </span><span style="color:#608b4e;">/// Send a remote command to the gRPC server
</span><span> #[structopt(setting </span><span style="color:#569cd6;">=</span><span> structopt::clap::AppSettings::TrailingVarArg)]
</span><span> Run(RemoteCommandOptions),
</span><span>}
</span></code></pre>
<ul>
<li>
<p>We’re working with an enum this time. But again, the most important part is the <code>derive(StructOpt)</code> attribute.</p>
</li>
<li>
<p>The reason to use an enum is to provide some development comfort. Each field in the enum takes in a struct where additional parsing occurs in the event that the subcommand is chosen. But this pattern enables us to not mix that up within this enum and make the code unfocused, and hard to read.</p>
</li>
</ul>
<hr />
<ul>
<li>
<p>The second most important detail is to notice the comments with 3 slashes <code>///</code>.</p>
</li>
<li>
<p>These are <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/reference/comments.html#doc-comments">doc comments</a>, and their placement is intentional. Rust will use these comments in the generated help command. The 2 slash comments are notes just for you, the developer, and are not seen by the user.</p>
</li>
</ul>
<hr />
<ul>
<li>For the first subcommand, admittedly I named this field <code>StartServer</code> so I could show off using the <code>structopt(name)</code> attribute.</li>
</ul>
<p>Without the attribute, the user would experience the subcommand transformed by default into the “kebab-case” form <code>start-command</code>. With the <code>name</code> defined on the StartServer field, we tell Rust that we want the user to use <code>server</code> instead.</p>
<p>(You can configure this behavior with the <code>structopt(rename_all)</code> attribute. I won’t be covering that. <a rel="noopener nofollow" target="_blank" href="https://docs.rs/structopt/0.3.16/structopt/#specifying-argument-types">Read more about rename_all in the docs</a>)</p>
<hr />
<p>The second subcommand <code>Run</code>... you’ll have to forgive my 👋hand waving👋.</p>
<ul>
<li>
<p>Remember that StructOpt is built on top of the <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/clap">Clap</a> crate.</p>
</li>
<li>
<p>Clap is quite flexible, but I thought it was much harder to use. StructOpt offers the ability to pass configuration to Clap and we’re setting a configuration setting w/ respect to the parsing behavior for only this subcommand.</p>
</li>
</ul>
<hr />
<ul>
<li>
<p>We want to pass a full command from the client to the server. But we don’t necessarily know how long that command will be and we don’t want the full command to be parsed.</p>
</li>
<li>
<p>The technical description for this kind of CLI parameter is a “Variable-length Argument” or a VarArg in this case. It is a hint for how to parse the last argument so you don’t need to define an end length -- it just trails off.</p>
</li>
<li>
<p>We are configuring the <code>Run</code> subcommand to tell Rust that this uses a VarArg. See <a rel="noopener nofollow" target="_blank" href="https://docs.rs/clap/2.33.1/clap/enum.AppSettings.html#variant.TrailingVarArg">the Clap docs</a> for more info about this and other AppSettings.</p>
</li>
</ul>
<h5 id="serveroptions">ServerOptions<a class="zola-anchor" href="#serveroptions" aria-label="Anchor link for: serveroptions">🔗</a></h5>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// These are the options used by the `server` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>ServerOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_listen_addr: String,
</span><span>}
</span></code></pre>
<ul>
<li>
<p>Our <code>server</code> subcommand has a single configurable option.</p>
</li>
<li>
<p>The <code>structopt(long)</code> attribute specifies that this is an option that the user will specify with the double-hyphen pattern with the name of the option, which will be in kebab-case by default. Therefore the user would use this as <code>--server-listen-addr</code>.</p>
</li>
<li>
<p><code>structopt(default_value)</code> is hopefully self-explanatory enough. If the user doesn’t override, the default value will be used. The default value type is a string slice <code>&str</code>, but structopt is converting it into a <code>String</code> by default.</p>
</li>
</ul>
<h5 id="remotecommandoptions">RemoteCommandOptions<a class="zola-anchor" href="#remotecommandoptions" aria-label="Anchor link for: remotecommandoptions">🔗</a></h5>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// These are the options used by the `run` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>RemoteCommandOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"http://127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_addr: String,
</span><span> </span><span style="color:#608b4e;">/// The full command and arguments for the server to execute
</span><span> </span><span style="color:#569cd6;">pub </span><span>command: Vec<String>,
</span><span>}
</span></code></pre>
<p>Our <code>run</code> subcommand has 2 possible arguments.</p>
<ol>
<li>The first, <code>server_addr</code> is an optional <code>structopt(long)</code> argument with a default value that aligns with the <code>server</code> default.</li>
<li>The second <code>command</code> is a required positional argument. Notice how there is no <code>structopt</code> attribute. The resulting vector from the variable-length argument. The parser splits up spaces per word, and provides them in order within the Vec<String>. (Matched quotes are interpreted as a single word in our situation). </li>
</ol>
<h5 id="main">main()<a class="zola-anchor" href="#main" aria-label="Anchor link for: main">🔗</a></h5>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>main() -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#569cd6;">let</span><span> args = ApplicationArguments::from_args();
</span><span>
</span><span> </span><span style="color:#569cd6;">match</span><span> args.subcommand {
</span><span> SubCommand::StartServer(opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Start the server on: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, opts.server_listen_addr);
</span><span> }
</span><span> SubCommand::Run(rc_opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Run command: '</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">'"</span><span>, rc_opts.command);
</span><span> }
</span><span> }
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>Our <code>main()</code> is short and focused.</p>
<ul>
<li>
<p>Our return type is a <code>Result</code>. We return <code>()</code> when things are good, and returns a boxed <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/reference/types/trait-object.html">trait object</a> that implements the <code>std::error::Error</code> trait as our error (the return trait object is boxed, because Rust doesn’t know how much space to allocate).</p>
</li>
<li>
<p>We parse the user input using our StructOpt customized <code>ApplicationArguments</code> struct with <code>from_args()</code>. What’s great is invalid inputs are handled, and so we don’t need to spend any time straying from the happy path.</p>
</li>
<li>
<p>After the parsing, we need to know what action to take next. We’ll either take a server action, or take a client action.</p>
</li>
<li>
<p>We pattern match on our <code>SubCommand</code> struct, and <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/rust-by-example/flow_control/match/destructuring/destructure_enum.html">destructure the enum’s internal structs</a> for the additional arguments.</p>
</li>
<li>
<p>We eventually will call out to the respective server or client to pass along the args. However for now we call <code>println!()</code> to display the values.</p>
</li>
</ul>
<h4 id="all-together">All together<a class="zola-anchor" href="#all-together" aria-label="Anchor link for: all-together">🔗</a></h4>
<p><strong>main.rs</strong></p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>structopt::StructOpt;
</span><span>
</span><span style="color:#608b4e;">// These are the options used by the `server` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>ServerOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_listen_addr: String,
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// These are the options used by the `run` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>RemoteCommandOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"http://127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_addr: String,
</span><span> </span><span style="color:#608b4e;">/// The full command and arguments for the server to execute
</span><span> </span><span style="color:#569cd6;">pub </span><span>command: Vec<String>,
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// These are the only valid values for our subcommands
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub enum </span><span>SubCommand {
</span><span> </span><span style="color:#608b4e;">/// Start the remote command gRPC server
</span><span> #[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>)]
</span><span> StartServer(ServerOptions),
</span><span> </span><span style="color:#608b4e;">/// Send a remote command to the gRPC server
</span><span> #[structopt(setting </span><span style="color:#569cd6;">=</span><span> structopt::clap::AppSettings::TrailingVarArg)]
</span><span> Run(RemoteCommandOptions),
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// This is the main arguments structure that we'll parse from
</span><span>#[derive(StructOpt, Debug)]
</span><span>#[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"remotecli"</span><span>)]
</span><span style="color:#569cd6;">struct </span><span>ApplicationArguments {
</span><span> #[structopt(flatten)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>subcommand: SubCommand,
</span><span>}
</span><span>
</span><span style="color:#569cd6;">fn </span><span>main() -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#569cd6;">let</span><span> args = ApplicationArguments::from_args();
</span><span>
</span><span> </span><span style="color:#569cd6;">match</span><span> args.subcommand {
</span><span> SubCommand::StartServer(opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Start the server on: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, opts.server_listen_addr);
</span><span> }
</span><span> SubCommand::Run(rc_opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Run command: '</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">'"</span><span>, rc_opts.command);
</span><span> }
</span><span> }
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>And that’s what we’ve done so far. This will be the full extent of the command line parsing functionality for this example, but we’ll revisit the <code>main()</code> function later.</p>
<p>If you’re following along, this code works with the <code>cargo.toml</code> provided at the top of this section. Play around using <code>cargo</code>.</p>
<p>For example try the following commands:</p>
<ul>
<li><code>cargo run --</code></li>
<li><code>cargo run -- server</code></li>
<li><code>cargo run -- server -h</code></li>
<li><code>cargo run -- run</code></li>
<li><code>cargo run -- run ls -al</code></li>
<li><code>cargo run -- run -h</code></li>
<li><code>cargo run -- blahblahblah</code></li>
</ul>
<h2 id="protocol-buffers">Protocol Buffers<a class="zola-anchor" href="#protocol-buffers" aria-label="Anchor link for: protocol-buffers">🔗</a></h2>
<div class="blog-image">
<img src="[object]" />
</div>
<h3 id="what-are-protocol-buffers">What are Protocol Buffers?<a class="zola-anchor" href="#what-are-protocol-buffers" aria-label="Anchor link for: what-are-protocol-buffers">🔗</a></h3>
<p><a rel="noopener nofollow" target="_blank" href="https://developers.google.com/protocol-buffers/docs/proto3">Protocol Buffers</a> (protobufs) are a way to define a data schema for how your data is structured as well as how to define how programs interface with each other w/ respect to your data in a language-independent manner.</p>
<p>This is achieved by writing your data in the protobuf format and compiling it into a supported language of your choice as implemented as <a rel="noopener nofollow" target="_blank" href="https://grpc.io/">gRPC</a>.</p>
<p>The result of the compilation generates a lot of boilerplate code.</p>
<p>Not just data structures with the same shape and naming conventions for your language’s native data types. But also generates the gRPC network code for the client that sends or the server that receives these generated data structures.</p>
<hr />
<p>For what it’s worth, an added bonus are servers and clients having the possibility to be implemented in different languages and inter-operate without issue due to. But we’re going to continue to work entirely in Rust for this example</p>
<h3 id="where-should-protobuf-live-in-the-codebase">Where should protobuf live in the codebase?<a class="zola-anchor" href="#where-should-protobuf-live-in-the-codebase" aria-label="Anchor link for: where-should-protobuf-live-in-the-codebase">🔗</a></h3>
<p>Before jumping into the protobuf, I wanted to mention my practice for where to keep the file itself.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ tree
</span><span>.
</span><span>├── Cargo.lock
</span><span>├── Cargo.toml
</span><span>├── proto
</span><span>│ └── cli.proto
</span><span>└── src
</span><span> └── main.rs
</span></code></pre>
<p>I like to keep the protobuf in a directory named <code>proto</code> typically at the same level as the <code>Cargo.toml</code> because as we’ll see soon, the build script will need to reference a path to the protobuf for compilation. The file name itself is arbitrary and <a rel="noopener nofollow" target="_blank" href="https://www.karlton.org/2017/12/naming-things-hard/">naming things is hard</a> so do your best to support your future self with meaningful names.</p>
<h3 id="the-example-protobuf">The example protobuf<a class="zola-anchor" href="#the-example-protobuf" aria-label="Anchor link for: the-example-protobuf">🔗</a></h3>
<h4 id="cli-proto">cli.proto<a class="zola-anchor" href="#cli-proto" aria-label="Anchor link for: cli-proto">🔗</a></h4>
<pre data-lang="proto" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-proto "><code class="language-proto" data-lang="proto"><span style="color:#569cd6;">syntax </span><span>= </span><span style="color:#d69d85;">"proto3"</span><span>;
</span><span>
</span><span style="color:#569cd6;">package </span><span>remotecli;
</span><span>
</span><span style="color:#608b4e;">// Command input
</span><span style="color:#569cd6;">message </span><span>CommandInput {
</span><span> string command = </span><span style="color:#b5cea8;">1</span><span>;
</span><span> </span><span style="color:#569cd6;">repeated </span><span>string args = </span><span style="color:#b5cea8;">2</span><span>;
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Command output
</span><span style="color:#569cd6;">message </span><span>CommandOutput {
</span><span> string output = </span><span style="color:#b5cea8;">1</span><span>;
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Service definition
</span><span style="color:#569cd6;">service </span><span>RemoteCLI {
</span><span> </span><span style="color:#569cd6;">rpc </span><span>Shell(CommandInput) </span><span style="color:#569cd6;">returns </span><span>(CommandOutput);
</span><span>}
</span></code></pre>
<p>We start the file off by declaring the particular version of syntax we’re using. <code>proto3</code>.</p>
<hr />
<ul>
<li>
<p>We need to provide a package name.</p>
</li>
<li>
<p>The <a rel="noopener nofollow" target="_blank" href="https://developers.google.com/protocol-buffers/docs/overview#packages">proto3 docs</a> say this is optional, but our protobuf Rust code generator <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/prost">Prost</a> requires it to be defined for module namespacing and naming the resulting file.</p>
</li>
</ul>
<hr />
<ul>
<li>
<p>Defined are 2 data structures, called <code>message</code>s.</p>
</li>
<li>
<p>The order of the fields are numbered and are important for identifying fields in the wire protocol when they are serialized/deserialized for gRPC communication.</p>
</li>
<li>
<p>The numbers in the message must be unique and the best practice is to not change the numbers once in use. </p>
</li>
</ul>
<p>(For more details, read more about Field numbers <a rel="noopener nofollow" target="_blank" href="https://developers.google.com/protocol-buffers/docs/proto3#assigning_field_numbers">in the docs</a>.)</p>
<hr />
<ul>
<li>
<p>The <code>CommandInput</code> message has 2 <code>string</code> fields - one singular and the other <code>repeated</code>. </p>
</li>
<li>
<p>The main executable, which we refer to as <code>command</code> the first word of the user input.</p>
</li>
<li>
<p>The rest of the user input is reserved for <code>args</code>.</p>
</li>
<li>
<p>The separation is meant to provide structure for the way a command interpreter like Bash defines commands.</p>
</li>
</ul>
<hr />
<ul>
<li>The <code>CommandOutput</code> message doesn’t need quite as much structure. After a command is run, the Standard Output will be returned as a single block of text.</li>
</ul>
<hr />
<ul>
<li>
<p>Finally, we define a service <code>RemoteCLI</code> with a single endpoint <code>Shell</code>.</p>
</li>
<li>
<p><code>Shell</code> takes a <code>CommandInput</code> and returns a <code>CommandOutput</code>.</p>
</li>
</ul>
<h3 id="compile-the-protobuf-into-rust-code-with-tonic">Compile the protobuf into Rust code with Tonic<a class="zola-anchor" href="#compile-the-protobuf-into-rust-code-with-tonic" aria-label="Anchor link for: compile-the-protobuf-into-rust-code-with-tonic">🔗</a></h3>
<div class="blog-image">
<img src="[object]" />
</div>
<p>Now that we have a protobuf, how do we use it in our Rust program when we need to use the generated code?</p>
<p>Well, we need to configure the build to compile the protobuf into Rust first.</p>
<p>The way we accomplish that is by using a <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/stable/rust-by-example/cargo/build_scripts.html">build script</a> (Surprise! Written in Rust) but is compiled and executed before the rest of the compilation occurs.</p>
<p>Cargo will run your build script if you have a file named <code>build.rs</code> in your project root.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ tree
</span><span>.
</span><span>├── build.rs
</span><span>├── Cargo.toml
</span><span>├── proto
</span><span>│ └── cli.proto
</span><span>└── src
</span><span> └── main.rs
</span></code></pre>
<h4 id="build-rs">build.rs<a class="zola-anchor" href="#build-rs" aria-label="Anchor link for: build-rs">🔗</a></h4>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>main() {
</span><span> tonic_build::compile_protos(</span><span style="color:#d69d85;">"proto/cli.proto"</span><span>).unwrap();
</span><span>}
</span></code></pre>
<p>The build script is just a small Rust program with a <code>main()</code> function.</p>
<p>We’re using <code>tonic_build</code> to compile our proto into Rust. We’ll see more <code>tonic</code> soon for the rest of our gRPC journey.</p>
<p>But for now we only need to add this crate into our <code>Cargo.toml</code> as a build dependency.</p>
<p><strong>Cargo.toml</strong></p>
<pre data-lang="toml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[</span><span style="color:#808080;">package</span><span>]
</span><span style="color:#569cd6;">name </span><span>= </span><span style="color:#d69d85;">"cli-grpc-tonic-blocking"
</span><span style="color:#569cd6;">version </span><span>= </span><span style="color:#d69d85;">"0.1.0"
</span><span style="color:#569cd6;">authors </span><span>= [</span><span style="color:#d69d85;">"T.J. Telan <t.telan@gmail.com>"</span><span>]
</span><span style="color:#569cd6;">edition </span><span>= </span><span style="color:#d69d85;">"2018"
</span><span>
</span><span>[</span><span style="color:#808080;">dependencies</span><span>]
</span><span style="color:#608b4e;"># CLI
</span><span style="color:#569cd6;">structopt </span><span>= </span><span style="color:#d69d85;">"0.3"
</span><span>
</span><span>[</span><span style="color:#808080;">build-dependencies</span><span>]
</span><span style="color:#608b4e;"># protobuf->Rust compiler
</span><span style="color:#569cd6;">tonic-build </span><span>= </span><span style="color:#d69d85;">"0.3.0"
</span></code></pre>
<p>Build dependencies are listed under its own section <code>[build-dependencies]</code>. If you didn’t know, your build scripts can only use crates listed in this section, and vice versa with the main package.</p>
<p>You can look at the resulting Rust code in your <code>target</code> directory when you <code>cargo build</code>.</p>
<p>You’ll have more than one directory with your package name plus extra generated characters due to build script output. So you may need to look through multiple directories.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ tree target/debug/build/cli-grpc-tonic-blocking-aa0556a3d0cd89ff/
</span><span>target/debug/build/cli-grpc-tonic-blocking-aa0556a3d0cd89ff/
</span><span>├── invoked.timestamp
</span><span>├── out
</span><span>│ └── remotecli.rs
</span><span>├── output
</span><span>├── root-output
</span><span>└── stderr
</span></code></pre>
<p>I’ll leave the contents of the generated code to those following along, since there’s a lot of it and the relevant info is either from the proto or will be covered in the server and client implementation.</p>
<p>This code will only generate once. Or unless you make changes to <code>build.rs</code>. So if you make changes to your proto and you want to regenerate code, you can force a code regen by using <code>touch</code>.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ touch build.rs
</span><span>$ cargo build
</span></code></pre>
<h2 id="server">Server<a class="zola-anchor" href="#server" aria-label="Anchor link for: server">🔗</a></h2>
<p>Moving onto writing our server now that we can use the protobuf generated code. We’re going to write the server (and client) in a new module.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ tree
</span><span>.
</span><span>├── build.rs
</span><span>├── Cargo.toml
</span><span>├── proto
</span><span>│ └── cli.proto
</span><span>└── src
</span><span> ├── main.rs
</span><span> └── remotecli
</span><span> ├── mod.rs
</span><span> └── server.rs
</span></code></pre>
<h3 id="cargo-toml">Cargo.toml<a class="zola-anchor" href="#cargo-toml" aria-label="Anchor link for: cargo-toml">🔗</a></h3>
<pre data-lang="toml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[</span><span style="color:#808080;">package</span><span>]
</span><span style="color:#569cd6;">name </span><span>= </span><span style="color:#d69d85;">"cli-grpc-tonic-blocking"
</span><span style="color:#569cd6;">version </span><span>= </span><span style="color:#d69d85;">"0.1.0"
</span><span style="color:#569cd6;">authors </span><span>= [</span><span style="color:#d69d85;">"T.J. Telan <t.telan@gmail.com>"</span><span>]
</span><span style="color:#569cd6;">edition </span><span>= </span><span style="color:#d69d85;">"2018"
</span><span>
</span><span>[</span><span style="color:#808080;">dependencies</span><span>]
</span><span style="color:#608b4e;"># gRPC server/client
</span><span style="color:#569cd6;">tonic </span><span>= </span><span style="color:#d69d85;">"0.3.0"
</span><span style="color:#569cd6;">prost </span><span>= </span><span style="color:#d69d85;">"0.6"
</span><span style="color:#608b4e;"># CLI
</span><span style="color:#569cd6;">structopt </span><span>= </span><span style="color:#d69d85;">"0.3"
</span><span style="color:#608b4e;"># Async runtime
</span><span style="color:#569cd6;">tokio </span><span>= { </span><span style="color:#569cd6;">version </span><span>= </span><span style="color:#d69d85;">"0.2"</span><span>, </span><span style="color:#569cd6;">features </span><span>= [</span><span style="color:#d69d85;">"full"</span><span>] }
</span><span>
</span><span>[</span><span style="color:#808080;">build-dependencies</span><span>]
</span><span style="color:#608b4e;"># protobuf->Rust compiler
</span><span style="color:#569cd6;">tonic-build </span><span>= </span><span style="color:#d69d85;">"0.3.0"
</span></code></pre>
<p><em>This is the last change we’ll be making to Cargo.toml.</em></p>
<p>We’re adding in <code>tonic</code> and <code>prost</code> as we implement the gRPC server/client. <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/prost">Prost</a> is the implementation of protocol buffers in Rust, and is needed to compile the generated code when we include it into the rest of the package.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://tokio.rs/">Tokio</a> is the async runtime we’re using. The gRPC server/client are <code>async</code> and we will need to adjust our <code>main()</code> to communicate more in the code that we’re now calling async functions..</p>
<h3 id="remotecli-mod-rs">remotecli/mod.rs<a class="zola-anchor" href="#remotecli-mod-rs" aria-label="Anchor link for: remotecli-mod-rs">🔗</a></h3>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub mod </span><span>server;
</span></code></pre>
<p>To keep the implementations organized, we’ll separate the server and client code further into their own modules. Starting with the server.</p>
<h3 id="remotecli-server-rs">remotecli/server.rs<a class="zola-anchor" href="#remotecli-server-rs" aria-label="Anchor link for: remotecli-server-rs">🔗</a></h3>
<p>Similar to the frontend CLI walkthrough, I’ll break this file up into pieces and review them.</p>
<p><strong>At the <a href="https://tjtelan.com/blog/lets-build-a-single-binary-grpc-server-client-with-rust-in-2020/#remotecli-server-rs-all-together">bottom of this file’s section</a> I’ll have the complete file there for copy/paste purposes.</strong></p>
<h4 id="imports">Imports<a class="zola-anchor" href="#imports" aria-label="Anchor link for: imports">🔗</a></h4>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>tonic::{transport::Server, Request, Response, Status};
</span><span>
</span><span style="color:#608b4e;">// Import the generated rust code into module
</span><span style="color:#569cd6;">pub mod </span><span>remotecli_proto {
</span><span> tonic::include_proto</span><span style="color:#569cd6;">!</span><span>(</span><span style="color:#d69d85;">"remotecli"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Proto generated server traits
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::remote_cli_server::{RemoteCli, RemoteCliServer};
</span><span>
</span><span style="color:#608b4e;">// Proto message structs
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::{CommandInput, CommandOutput};
</span><span>
</span><span style="color:#608b4e;">// For the server listening address
</span><span style="color:#569cd6;">use crate</span><span>::ServerOptions;
</span><span>
</span><span style="color:#608b4e;">// For executing commands
</span><span style="color:#569cd6;">use </span><span>std::process::{Command, Stdio};
</span></code></pre>
<ul>
<li>
<p>At the top of the file, we declare a module <code>remotecli_proto</code> that is intended to be scoped only in this file. The name <code>remotecli_proto</code> is arbitrary and for clarity purposes. </p>
</li>
<li>
<p>The <code>tonic::include_proto!()</code> macro effectively copy/pastes our protobuf translated Rust code (as per protobuf package name) into the module.</p>
</li>
</ul>
<hr />
<ul>
<li>
<p>The naming conventions of the protobuf translation can be a little confusing at first, but it is all consistent.</p>
</li>
<li>
<p>Our protobuf’s <code>RemoteCLI</code> service generates separate client and server modules using <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Snake_case">snake case</a> + <code>_server</code> or <code>_client</code>. While generated trait definitions use <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Camel_case">Pascal case</a> (a specific form of camel case with initial letter capitalized).</p>
</li>
<li>
<p>From the server specific generated code, we are importing a trait <code>RemoteCli</code> which requires that we implement our gRPC endpoint <code>Shell</code> with the same function signature.</p>
</li>
<li>
<p>Additionally we import <code>RemoteCliServer</code>, a generated server implementation that handles all the gRPC networking semantics but requires that we instantiate with a struct that implements the <code>RemoteCli</code> trait.</p>
</li>
</ul>
<hr />
<ul>
<li>
<p>The last import from the gRPC code are our protobuf messages <code>CommandInput</code> and <code>CommandOutput</code></p>
</li>
<li>
<p>From our frontend, we are importing the <code>ServerOptions</code> struct, since we are going to pass the user input in for the server listening address.</p>
</li>
</ul>
<hr />
<p>At last, we import from <code>std::process</code>. <code>Command</code> and <code>Stdio</code> - for executing commands and capturing output.</p>
<h4 id="remotecli-trait-implementation">RemoteCli Trait implementation<a class="zola-anchor" href="#remotecli-trait-implementation" aria-label="Anchor link for: remotecli-trait-implementation">🔗</a></h4>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[derive(Default)]
</span><span style="color:#569cd6;">pub struct </span><span>Cli {}
</span><span>
</span><span>#[tonic::async_trait]
</span><span style="color:#569cd6;">impl </span><span>RemoteCli </span><span style="color:#569cd6;">for </span><span>Cli {
</span><span> async </span><span style="color:#569cd6;">fn </span><span>shell(
</span><span> </span><span style="color:#569cd6;">&</span><span>self,
</span><span> request: Request<CommandInput>,
</span><span> ) -> Result<Response<CommandOutput>, Status> {
</span><span> </span><span style="color:#569cd6;">let</span><span> req_command = request.into_inner();
</span><span> </span><span style="color:#569cd6;">let</span><span> command = req_command.command;
</span><span> </span><span style="color:#569cd6;">let</span><span> args = req_command.args;
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"Running command: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;"> - args: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">&</span><span>command, </span><span style="color:#569cd6;">&</span><span>args);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> process = Command::new(command)
</span><span> .args(args)
</span><span> .stdout(Stdio::piped())
</span><span> .spawn()
</span><span> .expect(</span><span style="color:#d69d85;">"failed to execute child process"</span><span>);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> output = process
</span><span> .wait_with_output()
</span><span> .expect(</span><span style="color:#d69d85;">"failed to wait on child process"</span><span>);
</span><span> </span><span style="color:#569cd6;">let</span><span> output = output.stdout;
</span><span>
</span><span> Ok(Response::new(CommandOutput {
</span><span> output: String::from_utf8(output).unwrap(),
</span><span> }))
</span><span> }
</span><span>}
</span></code></pre>
<ul>
<li>
<p>We declare our own struct <code>Cli</code> because we need to <code>impl RemoteCli</code>.</p>
</li>
<li>
<p>Our generated code uses an <code>async</code> method. We add <code>#[tonic::async_trait]</code> to our trait impl so the server can use <code>async fn</code> on our method. We just have one method to define, <code>async fn shell()</code>.</p>
</li>
</ul>
<hr />
<ul>
<li>I’m 👋waving my hands👋 here for the function signature, but the way I initially learned how to write them was to go into the generated code, skimmed the code within the <code>remote_cli_server</code> module and modified the crate paths.</li>
</ul>
<hr />
<ul>
<li>
<p>The first thing we do when we enter <code>shell</code> is peel off the <code>tonic</code> wrapping from <code>request</code> with <code>.into_inner()</code>. We further separate the ownership of data into <code>command</code> and <code>args</code> vars.</p>
</li>
<li>
<p>We build out <code>process</code> as the <code>std::process::Command</code> struct so we can spawn the user’s process and capture stdout.</p>
</li>
<li>
<p>Then we wait for <code>process</code> to exit and collect the output with <code>.wait_with_output()</code>. We just want <code>stdout</code> so we further take ownership of just that handle.</p>
</li>
</ul>
<hr />
<ul>
<li>Last, we build a <code>tonic::Response</code>, converting the process stdout into a <code>String</code> while we instantiate our <code>CommandOutput</code>. Finally wrapping the <code>Response</code> in a <code>Result</code> and returning it to the client.</li>
</ul>
<h4 id="start-server">start_server<a class="zola-anchor" href="#start-server" aria-label="Anchor link for: start-server">🔗</a></h4>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub</span><span> async </span><span style="color:#569cd6;">fn </span><span>start_server(opts: ServerOptions) -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#569cd6;">let</span><span> addr = opts.server_listen_addr.parse().unwrap();
</span><span> </span><span style="color:#569cd6;">let</span><span> cli_server = Cli::default();
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"RemoteCliServer listening on </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">"</span><span>, addr);
</span><span>
</span><span> Server::builder()
</span><span> .add_service(RemoteCliServer::new(cli_server))
</span><span> .serve(addr)
</span><span> .await</span><span style="color:#569cd6;">?</span><span>;
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<ul>
<li>This function will be used by the frontend for the purpose of starting the server.</li>
</ul>
<hr />
<ul>
<li>The listening address is passed in through <code>opts</code>. It’s passed in as a <code>String</code>, but the compiler figures out what type we mean when we call <code>.parse()</code> due to how we use it later.</li>
</ul>
<hr />
<ul>
<li>We instantiate <code>cli_server</code> with the <code>Cli</code> struct which we implemented as the protobuf trait <code>RemoteCli</code>. </li>
</ul>
<hr />
<ul>
<li>
<p><code>tonic::Server::builder()</code> creates our gRPC server instance.</p>
</li>
<li>
<p>The <code>.add_service()</code> method takes <code>RemoteCliServer::new(cli_server)</code> to create a gRPC server with our generated endpoints via <code>RemoteCliServer</code> and our trait impl via <code>cli_server</code>.</p>
</li>
<li>
<p>The <code>serve()</code> method takes in our parsed listening address, providing the hint the compiler needed to infer the required type and returns an <code>async Result<T> </code> for us to <code>.await</code> on.</p>
</li>
</ul>
<h3 id="main-rs-so-far">main.rs - so far<a class="zola-anchor" href="#main-rs-so-far" aria-label="Anchor link for: main-rs-so-far">🔗</a></h3>
<p>We are making small changes to <code>main.rs</code> to plug in the server module. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub mod </span><span>remotecli;
</span><span>
</span><span style="color:#569cd6;">use </span><span>structopt::StructOpt;
</span><span>
</span><span style="color:#608b4e;">// These are the options used by the `server` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>ServerOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_listen_addr: String,
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// These are the options used by the `run` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>RemoteCommandOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"http://127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_addr: String,
</span><span> </span><span style="color:#608b4e;">/// The full command and arguments for the server to execute
</span><span> </span><span style="color:#569cd6;">pub </span><span>command: Vec<String>,
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// These are the only valid values for our subcommands
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub enum </span><span>SubCommand {
</span><span> </span><span style="color:#608b4e;">/// Start the remote command gRPC server
</span><span> #[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>)]
</span><span> StartServer(ServerOptions),
</span><span> </span><span style="color:#608b4e;">/// Send a remote command to the gRPC server
</span><span> #[structopt(setting </span><span style="color:#569cd6;">=</span><span> structopt::clap::AppSettings::TrailingVarArg)]
</span><span> Run(RemoteCommandOptions),
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// This is the main arguments structure that we'll parse from
</span><span>#[derive(StructOpt, Debug)]
</span><span>#[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"remotecli"</span><span>)]
</span><span style="color:#569cd6;">struct </span><span>ApplicationArguments {
</span><span> #[structopt(flatten)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>subcommand: SubCommand,
</span><span>}
</span><span>
</span><span>#[tokio::main]
</span><span>async </span><span style="color:#569cd6;">fn </span><span>main() -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#569cd6;">let</span><span> args = ApplicationArguments::from_args();
</span><span>
</span><span> </span><span style="color:#569cd6;">match</span><span> args.subcommand {
</span><span> SubCommand::StartServer(opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Start the server on: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, opts.server_listen_addr);
</span><span> remotecli::server::start_server(opts).await</span><span style="color:#569cd6;">?</span><span>;
</span><span> }
</span><span> SubCommand::Run(rc_opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Run command: '</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">'"</span><span>, rc_opts.command);
</span><span>
</span><span>
</span><span> }
</span><span> }
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<ul>
<li>
<p>We now import our <code>remotecli</code> module.</p>
</li>
<li>
<p>The <code>main()</code> function changes slightly as well. First, we change the function to be <code>async</code>.</p>
</li>
<li>
<p>We add the <code>#[tokio::main]</code> attribute to mark the async function for execution.</p>
</li>
<li>
<p>And we call our new <code>start_server()</code> to actually start a server when the user runs the <code>server</code> subcommand.</p>
</li>
</ul>
<h3 id="remotecli-server-rs-all-together">remotecli/server.rs all together<a class="zola-anchor" href="#remotecli-server-rs-all-together" aria-label="Anchor link for: remotecli-server-rs-all-together">🔗</a></h3>
<p>Here’s the final form of the server module.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>tonic::{transport::Server, Request, Response, Status};
</span><span>
</span><span style="color:#608b4e;">// Import the generated rust code into module
</span><span style="color:#569cd6;">pub mod </span><span>remotecli_proto {
</span><span> tonic::include_proto</span><span style="color:#569cd6;">!</span><span>(</span><span style="color:#d69d85;">"remotecli"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Proto generated server traits
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::remote_cli_server::{RemoteCli, RemoteCliServer};
</span><span>
</span><span style="color:#608b4e;">// Proto message structs
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::{CommandInput, CommandOutput};
</span><span>
</span><span style="color:#608b4e;">// For the server listening address
</span><span style="color:#569cd6;">use crate</span><span>::ServerOptions;
</span><span>
</span><span style="color:#608b4e;">// For executing commands
</span><span style="color:#569cd6;">use </span><span>std::process::{Command, Stdio};
</span><span>
</span><span>#[derive(Default)]
</span><span style="color:#569cd6;">pub struct </span><span>Cli {}
</span><span>
</span><span>#[tonic::async_trait]
</span><span style="color:#569cd6;">impl </span><span>RemoteCli </span><span style="color:#569cd6;">for </span><span>Cli {
</span><span> async </span><span style="color:#569cd6;">fn </span><span>shell(
</span><span> </span><span style="color:#569cd6;">&</span><span>self,
</span><span> request: Request<CommandInput>,
</span><span> ) -> Result<Response<CommandOutput>, Status> {
</span><span> </span><span style="color:#569cd6;">let</span><span> req_command = request.into_inner();
</span><span> </span><span style="color:#569cd6;">let</span><span> command = req_command.command;
</span><span> </span><span style="color:#569cd6;">let</span><span> args = req_command.args;
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"Running command: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;"> - args: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">&</span><span>command, </span><span style="color:#569cd6;">&</span><span>args);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> process = Command::new(command)
</span><span> .args(args)
</span><span> .stdout(Stdio::piped())
</span><span> .spawn()
</span><span> .expect(</span><span style="color:#d69d85;">"failed to execute child process"</span><span>);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> output = process
</span><span> .wait_with_output()
</span><span> .expect(</span><span style="color:#d69d85;">"failed to wait on child process"</span><span>);
</span><span> </span><span style="color:#569cd6;">let</span><span> output = output.stdout;
</span><span>
</span><span> Ok(Response::new(CommandOutput {
</span><span> output: String::from_utf8(output).unwrap(),
</span><span> }))
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#569cd6;">pub</span><span> async </span><span style="color:#569cd6;">fn </span><span>start_server(opts: ServerOptions) -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#569cd6;">let</span><span> addr = opts.server_listen_addr.parse().unwrap();
</span><span> </span><span style="color:#569cd6;">let</span><span> cli_server = Cli::default();
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"RemoteCliServer listening on </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">"</span><span>, addr);
</span><span>
</span><span> Server::builder()
</span><span> .add_service(RemoteCliServer::new(cli_server))
</span><span> .serve(addr)
</span><span> .await</span><span style="color:#569cd6;">?</span><span>;
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>And that’s the server implementation and the frontend code for starting the server. It is a surprisingly small amount of code.</p>
<hr />
<p>You can start an instance of the server by running:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo run</span><span style="color:#569cd6;"> --</span><span> server
</span><span>[...]
</span><span>Start the server on: </span><span style="color:#d69d85;">"127.0.0.1:50051"
</span><span>RemoteCliServer listening on 127.0.0.1:50051
</span></code></pre>
<h2 id="client">Client<a class="zola-anchor" href="#client" aria-label="Anchor link for: client">🔗</a></h2>
<p>We’re in the homestretch. Implementing a client. We’re going to create a new module within <code>remotecli</code> called <code>client.rs</code> that will follow the same patterns as we established for the server.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ tree
</span><span>.
</span><span>├── build.rs
</span><span>├── Cargo.toml
</span><span>├── proto
</span><span>│ └── cli.proto
</span><span>└── src
</span><span> ├── main.rs
</span><span> └── remotecli
</span><span> ├── client.rs
</span><span> ├── mod.rs
</span><span> └── server.rs
</span></code></pre>
<h3 id="remotecli-mod-rs-1">remotecli/mod.rs<a class="zola-anchor" href="#remotecli-mod-rs-1" aria-label="Anchor link for: remotecli-mod-rs-1">🔗</a></h3>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub mod </span><span>client;
</span><span style="color:#569cd6;">pub mod </span><span>server;
</span></code></pre>
<p>We’re declaring the client module within <code>mod.rs</code> </p>
<h3 id="remotecli-client-rs">remotecli/client.rs<a class="zola-anchor" href="#remotecli-client-rs" aria-label="Anchor link for: remotecli-client-rs">🔗</a></h3>
<p>Our client is a lot more straightforward. But splitting the module up into pieces for description purposes. </p>
<p><strong>Again, full file is at <a href="https://tjtelan.com/blog/lets-build-a-single-binary-grpc-server-client-with-rust-in-2020/#remotecli-client-rs-all-together">the end of the section</a></strong></p>
<h4 id="imports-1">Imports<a class="zola-anchor" href="#imports-1" aria-label="Anchor link for: imports-1">🔗</a></h4>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub mod </span><span>remotecli_proto {
</span><span> tonic::include_proto</span><span style="color:#569cd6;">!</span><span>(</span><span style="color:#d69d85;">"remotecli"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Proto generated client
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::remote_cli_client::RemoteCliClient;
</span><span>
</span><span style="color:#608b4e;">// Proto message structs
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::CommandInput;
</span><span>
</span><span style="color:#569cd6;">use crate</span><span>::RemoteCommandOptions;
</span></code></pre>
<ul>
<li>
<p>Just like in our server, we create a module <code>remotecli_proto</code> and we use the <code>tonic::include_proto!()</code> macro to copy/paste our generated code into this module.</p>
</li>
<li>
<p>We then include the generated <code>RemoteCliClient</code> to connect, and the <code>CommandInput</code> struct since that is what we send over to the server.</p>
</li>
<li>
<p>Last include is the <code>RemoteCommandOptions</code> struct from the frontend so we can pass in the server address we want to connect to.</p>
</li>
</ul>
<h4 id="client-run">client_run<a class="zola-anchor" href="#client-run" aria-label="Anchor link for: client-run">🔗</a></h4>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub</span><span> async </span><span style="color:#569cd6;">fn </span><span>client_run(rc_opts: RemoteCommandOptions) -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#608b4e;">// Connect to server
</span><span> </span><span style="color:#608b4e;">// Use server addr if given, otherwise use default
</span><span> </span><span style="color:#569cd6;">let mut</span><span> client = RemoteCliClient::connect(rc_opts.server_addr).await</span><span style="color:#569cd6;">?</span><span>;
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> request = tonic::Request::new(CommandInput {
</span><span> command: rc_opts.command[</span><span style="color:#b5cea8;">0</span><span>].clone().into(),
</span><span> args: rc_opts.command[</span><span style="color:#b5cea8;">1</span><span style="color:#569cd6;">..</span><span>].to_vec(),
</span><span> });
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> response = client.shell(request).await</span><span style="color:#569cd6;">?</span><span>;
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"RESPONSE=</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, response);
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<ul>
<li>The helper function <code>client_run()</code> is an <code>async</code> function like our server. The frontend passes in a <code>RemoteCommandOptions</code> struct for the server address info as well as our raw user command.</li>
</ul>
<hr />
<ul>
<li>First thing we do is create <code>client</code> and connect to the server with <code>RemoteCliClient::connect</code> and do an <code>.await</code>.</li>
</ul>
<hr />
<ul>
<li>
<p>Then we build our request by creating a <code>tonic::Request</code> struct with our <code>CommandInput</code>.</p>
</li>
<li>
<p>The user command is raw and needs to be sliced up to fit the shape of what the server expects. The first word of the user command is the shell command, and the rest are the arguments.</p>
</li>
</ul>
<hr />
<ul>
<li>Lastly we use <code>client</code> and call our endpoint with our request and <code>.await</code> for the execution to complete.</li>
</ul>
<h3 id="main-rs">main.rs<a class="zola-anchor" href="#main-rs" aria-label="Anchor link for: main-rs">🔗</a></h3>
<p>This is the final form of <code>main.rs</code>. The last thing we do to <code>main.rs</code> is plug in our <code>client_run()</code> function.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub mod </span><span>remotecli;
</span><span>
</span><span style="color:#569cd6;">use </span><span>structopt::StructOpt;
</span><span>
</span><span style="color:#608b4e;">// These are the options used by the `server` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>ServerOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_listen_addr: String,
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// These are the options used by the `run` subcommand
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub struct </span><span>RemoteCommandOptions {
</span><span> </span><span style="color:#608b4e;">/// The address of the server that will run commands.
</span><span> #[structopt(long </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>, default_value </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"http://127.0.0.1:50051"</span><span>)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>server_addr: String,
</span><span> </span><span style="color:#608b4e;">/// The full command and arguments for the server to execute
</span><span> </span><span style="color:#569cd6;">pub </span><span>command: Vec<String>,
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// These are the only valid values for our subcommands
</span><span>#[derive(Debug, StructOpt)]
</span><span style="color:#569cd6;">pub enum </span><span>SubCommand {
</span><span> </span><span style="color:#608b4e;">/// Start the remote command gRPC server
</span><span> #[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"server"</span><span>)]
</span><span> StartServer(ServerOptions),
</span><span> </span><span style="color:#608b4e;">/// Send a remote command to the gRPC server
</span><span> #[structopt(setting </span><span style="color:#569cd6;">=</span><span> structopt::clap::AppSettings::TrailingVarArg)]
</span><span> Run(RemoteCommandOptions),
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// This is the main arguments structure that we'll parse from
</span><span>#[derive(StructOpt, Debug)]
</span><span>#[structopt(name </span><span style="color:#569cd6;">= </span><span style="color:#d69d85;">"remotecli"</span><span>)]
</span><span style="color:#569cd6;">struct </span><span>ApplicationArguments {
</span><span> #[structopt(flatten)]
</span><span> </span><span style="color:#569cd6;">pub </span><span>subcommand: SubCommand,
</span><span>}
</span><span>
</span><span>#[tokio::main]
</span><span>async </span><span style="color:#569cd6;">fn </span><span>main() -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#569cd6;">let</span><span> args = ApplicationArguments::from_args();
</span><span>
</span><span> </span><span style="color:#569cd6;">match</span><span> args.subcommand {
</span><span> SubCommand::StartServer(opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Start the server on: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, opts.server_listen_addr);
</span><span> remotecli::server::start_server(opts).await</span><span style="color:#569cd6;">?</span><span>;
</span><span> }
</span><span> SubCommand::Run(rc_opts) </span><span style="color:#569cd6;">=> </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"Run command: '</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">'"</span><span>, rc_opts.command);
</span><span> remotecli::client::client_run(rc_opts).await</span><span style="color:#569cd6;">?</span><span>;
</span><span> }
</span><span> }
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<h3 id="remotecli-client-rs-all-together">remotecli/client.rs all together<a class="zola-anchor" href="#remotecli-client-rs-all-together" aria-label="Anchor link for: remotecli-client-rs-all-together">🔗</a></h3>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub mod </span><span>remotecli_proto {
</span><span> tonic::include_proto</span><span style="color:#569cd6;">!</span><span>(</span><span style="color:#d69d85;">"remotecli"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Proto generated client
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::remote_cli_client::RemoteCliClient;
</span><span>
</span><span style="color:#608b4e;">// Proto message structs
</span><span style="color:#569cd6;">use </span><span>remotecli_proto::CommandInput;
</span><span>
</span><span style="color:#569cd6;">use crate</span><span>::RemoteCommandOptions;
</span><span>
</span><span style="color:#569cd6;">pub</span><span> async </span><span style="color:#569cd6;">fn </span><span>client_run(rc_opts: RemoteCommandOptions) -> Result<(), Box<dyn std::error::Error>> {
</span><span> </span><span style="color:#608b4e;">// Connect to server
</span><span> </span><span style="color:#608b4e;">// Use server addr if given, otherwise use default
</span><span> </span><span style="color:#569cd6;">let mut</span><span> client = RemoteCliClient::connect(rc_opts.server_addr).await</span><span style="color:#569cd6;">?</span><span>;
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> request = tonic::Request::new(CommandInput {
</span><span> command: rc_opts.command[</span><span style="color:#b5cea8;">0</span><span>].clone().into(),
</span><span> args: rc_opts.command[</span><span style="color:#b5cea8;">1</span><span style="color:#569cd6;">..</span><span>].to_vec(),
</span><span> });
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> response = client.shell(request).await</span><span style="color:#569cd6;">?</span><span>;
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"RESPONSE=</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, response);
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<h3 id="final-demonstration">Final demonstration<a class="zola-anchor" href="#final-demonstration" aria-label="Anchor link for: final-demonstration">🔗</a></h3>
<p>To see this server-client end-to-end, we'll need two terminal windows open. In one, run the server, and in the other we'll run a simple <code>ls</code> command.</p>
<h4 id="server-1">Server<a class="zola-anchor" href="#server-1" aria-label="Anchor link for: server-1">🔗</a></h4>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo run</span><span style="color:#569cd6;"> --</span><span> server
</span><span>[...]
</span><span>Start the server on: </span><span style="color:#d69d85;">"127.0.0.1:50051"
</span><span>RemoteCliServer listening on 127.0.0.1:50051
</span></code></pre>
<h4 id="client-1">Client<a class="zola-anchor" href="#client-1" aria-label="Anchor link for: client-1">🔗</a></h4>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ cargo run</span><span style="color:#569cd6;"> --</span><span> run ls
</span></code></pre>
<h4 id="output">Output<a class="zola-anchor" href="#output" aria-label="Anchor link for: output">🔗</a></h4>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>Run command: </span><span style="color:#d69d85;">'["ls"]'
</span><span>RESPONSE=</span><span style="background-color:#282828;color:#d69d85;">Response</span><span> { metadata: MetadataMap { headers: {</span><span style="color:#d69d85;">"content-type"</span><span>: </span><span style="color:#d69d85;">"application/grpc"</span><span>, </span><span style="color:#d69d85;">"date"</span><span>: </span><span style="color:#d69d85;">"Wed, 19 Aug 2020 00:00:25 GMT"</span><span>, </span><span style="color:#d69d85;">"grpc-status"</span><span>: </span><span style="color:#d69d85;">"0"</span><span>} }, message: CommandOutput { output: </span><span style="color:#d69d85;">"build.rs\nCargo.toml\nproto\nsrc\n" </span><span>} }
</span></code></pre>
<p>As we see, there is still work left to do in order to format the output in a more human readable way. But that is an exercise left to the reader.</p>
<h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2>
<p>We just walked through building a CLI application that parses user input and uses gRPC to send a command from a gRPC client to the server for execution and return of command output.</p>
<p>Based on how we structured the frontend CLI using <code>StructOpt</code>, we allowed both the client and server to compile into a single binary. </p>
<p>Protocol buffers (or protobufs) were used to define the interfaces of the server and the data structures that were used. The <code>Tonic</code> and <code>Prost</code> crates and Cargo build scripts were used to compile the protobufs into native async Rust code.</p>
<p><code>Tokio</code> was our async runtime. We experienced how little code was necessary to support <code>async</code>/<code>await</code> patterns.</p>
<p>Thanks for joining me as I stepped through the details. I hope that this walkthrough satisfies some curiosity about using gRPC for your backend code. As well as piqued your interest in writing some Rust code.</p>
What I learned running a live programming Twitch stream from Linux2020-07-29T00:00:00+00:002020-08-03T00:00:00+00:00https://tjtelan.com/blog/what-i-learned-running-a-live-programming-stream-from-linux/<p>I recently completed my first coding streaming session. A few people watched for a few minutes, so I considered it to be a success. I used what I already had on hand, did not spend any money and I was able to stream for about 3 hours from zero code to a small completed project. There is much room for improvement, but I want to share what I used to get started. </p>
<p>My motivation for streaming my coding sessions is to improve my communication skills, as well as educate and demonstrate to others about being productive with programming languages and tech that I like using.</p>
<h2 id="my-streaming-setup">My streaming setup<a class="zola-anchor" href="#my-streaming-setup" aria-label="Anchor link for: my-streaming-setup">🔗</a></h2>
<p>I’m running Linux on my <a rel="noopener nofollow" target="_blank" href="https://system76.com/guides/gaze14/15b">System76 Gazelle</a> laptop. My distro is Manjaro 20 (KDE Plasma).</p>
<p>The following are the hardware + software I already had</p>
<ul>
<li>Laptop webcam</li>
<li>Condenser microphone (I use a Behringer C-1U)</li>
<li>Adjustable mic stand</li>
<li>External Monitor</li>
<li>External keyboard</li>
<li>External mouse</li>
<li>Streaming software (I used OBS. More on that later)</li>
<li>Text editor (I used VSCode)</li>
<li>Terminal (I used Alacritty + the built-in terminal in VSCode)</li>
<li>Web browser</li>
</ul>
<div class="blog-image"><figure>
<img src="[object]" alt="My extra basic streaming setup. Don't spend big money before you even have momentum." /><figcaption>My extra basic streaming setup. Don't spend big money before you even have momentum.</figcaption></figure>
</div>
<h2 id="set-accurate-expectations-for-yourself-and-potential-viewers">Set accurate expectations for yourself and potential viewers<a class="zola-anchor" href="#set-accurate-expectations-for-yourself-and-potential-viewers" aria-label="Anchor link for: set-accurate-expectations-for-yourself-and-potential-viewers">🔗</a></h2>
<p>I watched a few coding streams from people that I see in the space regularly prior to running my own. I didn't want to reinvent the wheel, and so I had to experience what works for others and identify anything that is common. Pretty much <em>no one</em> is doing anything wholly unique, so I decided I would learn by copying what works for other coding streamers.</p>
<h3 id="create-loose-structure-to-guide-you">Create loose structure to guide you<a class="zola-anchor" href="#create-loose-structure-to-guide-you" aria-label="Anchor link for: create-loose-structure-to-guide-you">🔗</a></h3>
<h4 id="write-your-title-with-intention">Write your title with intention<a class="zola-anchor" href="#write-your-title-with-intention" aria-label="Anchor link for: write-your-title-with-intention">🔗</a></h4>
<p>Unless you have an existing community, try to avoid low-value clickbait.</p>
<p>As a lowly unknown I need to bring a simple, but descriptive stream title. </p>
<p>I want people to know:</p>
<ul>
<li>What language I'm writing in.</li>
<li>What <em>fancy technology</em> I plan to use, if any.</li>
<li>Succinct objective (Emphasis on <strong>singular</strong>!)</li>
</ul>
<p><em>Something to keep in mind</em>: Titles get truncated at about 40 or so characters. After that, viewers will have to put in some effort to mouse over your title to read past the cut.</p>
<blockquote>
<p>The title I settled on?</p>
<p>[Rust] GRPC Remote Command Line using Tonic + CLAP - Writing a blog post</p>
</blockquote>
<h4 id="add-a-description-and-have-a-plan">Add a description and have a plan<a class="zola-anchor" href="#add-a-description-and-have-a-plan" aria-label="Anchor link for: add-a-description-and-have-a-plan">🔗</a></h4>
<p>Most of the streamers have some narrow theme for their time online in order to set expectations for people watching.</p>
<p>Explicitly let your viewers know what they are going to see in your description so they can decide if they will stick around. Otherwise you'll either get asked the same questions or people will just bounce out with disinterest.</p>
<blockquote>
<p>Specific to my first stream:</p>
<p>I wanted to dedicate time to writing the example Rust code for another blog post I have in progress.</p>
<p>With that in mind:</p>
<p><strong>Description</strong></p>
<ul>
<li>Primary goal is to write a small CLI driven gRPC server/client in Rust</li>
<li>Secondary goal is to document my process including looking up documentation and testing.</li>
</ul>
<p><strong>Plan</strong></p>
<ul>
<li>In my notes, I broke down the domains of the application into ordered objectives I could follow.</li>
<li>I would take on each domain one at a time with the intention of connecting each of them together as each section became stable enough to test with user input.</li>
</ul>
</blockquote>
<h4 id="technical-preparation">Technical preparation<a class="zola-anchor" href="#technical-preparation" aria-label="Anchor link for: technical-preparation">🔗</a></h4>
<p>Since I didn’t want to worry about getting everything perfect I narrowed my focus on a few details and personal preferences about the stream.</p>
<ul>
<li>Not streaming my entire desktop, instead individual windows.
<ul>
<li>(I thought this would be easier for viewers.)</li>
</ul>
</li>
<li>Including a background for the negative space</li>
<li>Small on-screen banner that includes links to my other online platforms
<ul>
<li>Minor self-promotion</li>
</ul>
</li>
<li>Microphone only, no external music</li>
<li>Webcam placement as close to level to or pointing down on my face
<ul>
<li>So people wouldn't have to look up my nose</li>
</ul>
</li>
</ul>
<p>I didn't worry about having chat visible but I did have it open for me to interact with just in case.</p>
<h2 id="set-up-streaming-software">Set up streaming software<a class="zola-anchor" href="#set-up-streaming-software" aria-label="Anchor link for: set-up-streaming-software">🔗</a></h2>
<p>Twitch has a list of <a rel="noopener nofollow" target="_blank" href="https://help.twitch.tv/s/article/recommended-software-for-broadcasting?language=en_US">recommended software for broadcasting</a>. I ended up using Open Broadcaster Software, more commonly known as OBS because it has support for Linux.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="Open Broadcaster Software a.k.a OBS" /><figcaption>Open Broadcaster Software a.k.a OBS</figcaption></figure>
</div>
<p>I’m not going to go over the deep details of OBS, but I encourage looking for tutorials on Youtube. Even in 2020, the beginner OBS tutorials from a few years ago are still relevant and usable. They helped make using OBS a lot less intimidating.</p>
<p>Enable the preview and prepare to spend a little bit of time in OBS. What you see in the preview is what will get streamed out. Start simple. One Scene, and start with one source and add in sources as you get more familiar.</p>
<h3 id="quick-start-guide-to-sources">Quick start guide to Sources<a class="zola-anchor" href="#quick-start-guide-to-sources" aria-label="Anchor link for: quick-start-guide-to-sources">🔗</a></h3>
<p>Here’s a short list of sources you may want to add from a Linux desktop into OBS and what they translate to to usable in OBS.</p>
<table><thead><tr><th>Description of input</th><th>OBS source</th><th>Notes</th></tr></thead><tbody>
<tr><td>Entire desktop</td><td>Screen Capture (XSHM)</td><td>If you have multiple monitors, you will need to select the screen you want to display</td></tr>
<tr><td>Specific window</td><td>Window Capture (XComposite)</td><td>You’ll need to select the window you want displayed. One source per window</td></tr>
<tr><td>Image</td><td>Image</td><td>Tested w/ JPEG, PNG</td></tr>
<tr><td>Webcam</td><td>Video Capture Device (V4L2)</td><td>You’ll need to select your webcam from Devices if you have multiple cameras attached</td></tr>
<tr><td>Capture Card</td><td>Video Capture Device (V4L2)</td><td>Similar to webcam, you’ll need to select your capture card device. Especially if you also have a webcam attached to your computer</td></tr>
<tr><td>Microphone</td><td>- Audio Capture Device (ALSA)<br />- Audio Capture Input (PulseAudio)</td><td>It might be easier to configure audio through: Settings > Audio > Devices Especially if you want to disable sources</td></tr>
<tr><td>Desktop audio</td><td>- Audio Capture Device (ALSA)<br />- Audio Capture Input (PulseAudio)</td><td>(See notes for microphone)</td></tr>
</tbody></table>
<p>This may slightly differ from what the <a rel="noopener nofollow" target="_blank" href="https://obsproject.com/wiki/Sources-Guide">official OBS Sources guide</a> documents, but if you’re choosing to use Linux to stream then you probably already expect this. </p>
<div class="blog-image"><figure>
<img src="[object]" alt="What OBS sources look like in Linux" /><figcaption>What OBS sources look like in Linux</figcaption></figure>
</div>
<p>The <code>Sources > Add</code> drop-down menu has icons that give good hints for what they do. Your results may vary, so be prepared to play around with settings. (But you already know that if you're insisting on using Linux...)</p>
<h4 id="workarounds-for-blacked-out-screens-using-laptop-hybrid-gpu-setups">Workarounds for blacked out screens using laptop hybrid GPU setups<a class="zola-anchor" href="#workarounds-for-blacked-out-screens-using-laptop-hybrid-gpu-setups" aria-label="Anchor link for: workarounds-for-blacked-out-screens-using-laptop-hybrid-gpu-setups">🔗</a></h4>
<p>One issue I ran into while writing his guide was being unable to use a single window as a streaming source. Adding the source resulted in a window that was blacked out, but my mouse cursor was still visible when over the window.</p>
<div class="blog-image"><figure>
<img src="[object]" alt="VSCode's window is blacked out in OBS, but my cursor is still visible..." /><figcaption>VSCode's window is blacked out in OBS, but my cursor is still visible...</figcaption></figure>
</div>
<p>To work around, you can pick one of the following solutions:</p>
<ul>
<li>Force OBS and all your windows to run on the same GPU. Some software such as web browsers or IDEs use hardware acceleration by default. You can enable/disable accordingly to match OBS.</li>
<li>Or you can restart your laptop in either integrated or GPU mode (i.e., not hybrid mode)</li>
</ul>
<p>The reason is due to how hybrid GPU works. OBS and the window I was trying to display were running on different gpus. </p>
<p>For more information please read <a rel="noopener nofollow" target="_blank" href="https://obsproject.com/forum/threads/laptop-black-screen-when-capturing-read-here-first.5965/">this thread from the OBS Project forums</a>.</p>
<h3 id="configuring-api-keys-for-streaming">Configuring API keys for streaming<a class="zola-anchor" href="#configuring-api-keys-for-streaming" aria-label="Anchor link for: configuring-api-keys-for-streaming">🔗</a></h3>
<div class="blog-image"><figure>
<img src="[object]" alt="OBS stream key settings" /><figcaption>OBS stream key settings</figcaption></figure>
</div>
<p>The location for where you add in your service stream key is very easy to navigate to:</p>
<p><code>Settings > Stream</code></p>
<p>Select the service you want to stream to, and enter the Stream Key.</p>
<p>There are many preconfigured services in OBS, so you’ll probably only need to provide your stream key and not the service URL.</p>
<p>Common links to where you can get your stream key:</p>
<ul>
<li>Youtube
<ol>
<li><a rel="noopener nofollow" target="_blank" href="https://www.youtube.com/live_dashboard">https://www.youtube.com/live_dashboard</a></li>
<li><code>Stream name/key</code> is at the button of the page</li>
</ol>
</li>
<li>Twitch
<ol>
<li><a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/settings/profile">https://www.twitch.tv/settings/profile</a> -- Then click <code>Channels and Videos</code></li>
<li>Or: <a rel="noopener nofollow" target="_blank" href="https://dashboard.twitch.tv/u/">https://dashboard.twitch.tv/u/</a><your-twitch-username>/settings/channel</li>
<li><code>Primary Stream Key</code> is listed at the top of the page</li>
</ol>
</li>
</ul>
<p>(I use a service called <a rel="noopener nofollow" target="_blank" href="https://restream.io/">Restream</a> which is supported by OBS. Restream will broadcast to multiple services at the same time. I use it to broadcast to YouTube and Twitch simultaneously.)</p>
<h2 id="don-t-overthink-and-start-streaming">Don’t overthink and start streaming<a class="zola-anchor" href="#don-t-overthink-and-start-streaming" aria-label="Anchor link for: don-t-overthink-and-start-streaming">🔗</a></h2>
<div class="blog-image"><figure>
<img src="[object]" alt="Don't worry about perfection. Just take the first step and start." /><figcaption>Don't worry about perfection. Just take the first step and start.</figcaption></figure>
</div>
<p>The first thing I noticed after going live was that the platforms I was broadcasting to had some noticeable lag between my actions and seeing it live in the browser.</p>
<p>When you first get started, you probably won’t have a lot of people watching. This will be a good thing because you’re going to notice many details you’ll want to improve. </p>
<p>I'm saying this to myself for the benefit of anyone else who got this far:</p>
<p>It will take time, effort and consistency on your part before you see results. So relax, try to have fun and enjoy the process.</p>
<hr />
<p>If you made this far, thanks!</p>
<p>Please <a rel="noopener nofollow" target="_blank" href="https://www.twitch.tv/tjtelan">check me out on my Twitch channel</a>. I stream weekly coding sessions about using Rust and DevOps topics. I hope you'll considering following me!</p>
How to prevent Github Actions from deploying on PR with CI/CD2020-06-12T00:00:00+00:002020-10-21T00:00:00+00:00https://tjtelan.com/blog/github-actions-push-vs-pr-workflow/<h2 id="my-experience-using-github-actions-for-ci-cd-as-a-solo-contributor">My experience using Github Actions for CI/CD as a solo contributor<a class="zola-anchor" href="#my-experience-using-github-actions-for-ci-cd-as-a-solo-contributor" aria-label="Anchor link for: my-experience-using-github-actions-for-ci-cd-as-a-solo-contributor">🔗</a></h2>
<p>I am using Github Actions to build and deploy my website when I push. That is a classic continuous integration / continuous deployment workflow. It’s convenient to commit, push and have my site build and deploy as a result. This workflow is simple but only works because I am the only contributor.</p>
<h2 id="github-actions-for-ci-cd-with-pull-request">Github Actions for CI/CD with Pull Request<a class="zola-anchor" href="#github-actions-for-ci-cd-with-pull-request" aria-label="Anchor link for: github-actions-for-ci-cd-with-pull-request">🔗</a></h2>
<p>It is a good practice to run sanity checks on pull requests prior to merging. How would that be accomplished with Github Actions?</p>
<p>It turns out that you can do this but you need to be very intentional with how your jobs are configured.</p>
<p>In my workflow, I am designating a main branch <code>main</code> that will run full CI/CD. Build, test and deploy. And for any other branch, just build and test.</p>
<p>I’ll share my example Github Actions workflow file, then I’ll provide a template that you can modify and use for your own purposes.</p>
<h2 id="my-example-github-actions-workflow">My example Github Actions workflow<a class="zola-anchor" href="#my-example-github-actions-workflow" aria-label="Anchor link for: my-example-github-actions-workflow">🔗</a></h2>
<p>Here’s my site’s current workflow file for Github Actions. I’ll break this down.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#569cd6;">on</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">push</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">branches</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pull_request</span><span>:
</span><span style="background-color:#282828;color:#569cd6;">jobs</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">build</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref != 'refs/heads/main'</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Build only'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">tjtelan/zola-deploy-action@main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">BUILD_DIR</span><span>: </span><span style="color:#b5cea8;">.
</span><span> </span><span style="background-color:#282828;color:#569cd6;">TOKEN</span><span>: </span><span style="background-color:#282828;color:#d69d85;">${{ secrets.TOKEN }}</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">BUILD_ONLY</span><span>: </span><span style="color:#569cd6;">true
</span><span> </span><span style="background-color:#282828;color:#569cd6;">build_and_deploy</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref == 'refs/heads/main'</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Build and deploy'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">tjtelan/zola-deploy-action@main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">PAGES_BRANCH</span><span>: </span><span style="background-color:#282828;color:#d69d85;">gh-pages</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">BUILD_DIR</span><span>: </span><span style="color:#b5cea8;">.
</span><span> </span><span style="background-color:#282828;color:#569cd6;">TOKEN</span><span>: </span><span style="background-color:#282828;color:#d69d85;">${{ secrets.TOKEN }}</span><span>
</span></code></pre>
<p>At the top, I am specifying the events that I want to trigger on with the <code>on</code> top-level key.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#569cd6;">on</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">push</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">branches</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pull_request</span><span>:
</span></code></pre>
<p>I want to trigger on push events to the <code>main</code> branch, and all pull requests.</p>
<p>Later below are 2 jobs that are almost identical. I’ll break them down one at a time then compare their differences.</p>
<p><code>build</code> job </p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span> </span><span style="background-color:#282828;color:#569cd6;">build</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref != 'refs/heads/main'</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Build only'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">tjtelan/zola-deploy-action@main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">BUILD_DIR</span><span>: </span><span style="color:#b5cea8;">.
</span><span> </span><span style="background-color:#282828;color:#569cd6;">TOKEN</span><span>: </span><span style="background-color:#282828;color:#d69d85;">${{ secrets.TOKEN }}</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">BUILD_ONLY</span><span>: </span><span style="color:#569cd6;">true
</span></code></pre>
<ol>
<li>This job uses the <code>ubuntu-latest</code> github hosted runner as my environment.</li>
<li>I do a check for the git ref via the <code>github.ref</code> key. Or another way to say this is I check that the working branch that triggered this job is not the <code>main</code> branch. I’ll continue forward only if this condition is <code>true</code>.</li>
<li>Lastly are my steps. I use the <code>actions/checkout@main</code> marketplace action to check my code out, and I use my fork of an action for Zola called <code>tjtelan/zola-deploy-action@main</code>. I have an environment variable <code>BUILD_ONLY</code> set to <code>true</code> which results in building my site but not deploying it.</li>
</ol>
<p><code>build_and_deploy</code> job</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span> </span><span style="background-color:#282828;color:#569cd6;">build_and_deploy</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref == 'refs/heads/main'</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Build and deploy'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">tjtelan/zola-deploy-action@main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">PAGES_BRANCH</span><span>: </span><span style="background-color:#282828;color:#d69d85;">gh-pages</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">BUILD_DIR</span><span>: </span><span style="color:#b5cea8;">.
</span><span> </span><span style="background-color:#282828;color:#569cd6;">TOKEN</span><span>: </span><span style="background-color:#282828;color:#d69d85;">${{ secrets.TOKEN }}</span><span>
</span></code></pre>
<ol>
<li>This job also uses the <code>ubuntu-latest</code> github hosted runner as my environment.</li>
<li>I do a similar check for the git ref via the <code>github.ref</code> key. This time I am looking for the working branch to be <code>main</code></li>
<li>Lastly are my steps. Same as the previous job, but I am configuring <code>tjtelan/zola-deploy-action@main</code> differently. Rather than setting <code>BUILD_ONLY</code>, I am defining <code>PAGES_BRANCH</code> to <code>gh-pages</code>, which is where I want to deploy my site code after build.</li>
</ol>
<h2 id="last-words-on-job-differences">Last words on job differences<a class="zola-anchor" href="#last-words-on-job-differences" aria-label="Anchor link for: last-words-on-job-differences">🔗</a></h2>
<p>The only differences are the branch check via <code>github.ref</code>, and the specifics of <code>steps</code>. I happen to be using my own Github Action <code>tjtelan/zola-deploy-action</code> but your steps could consist of anything you want to do differently between pull requests and push to your “special” branch.</p>
<h2 id="github-actions-template">Github Actions template<a class="zola-anchor" href="#github-actions-template" aria-label="Anchor link for: github-actions-template">🔗</a></h2>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#569cd6;">on</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">push</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">branches</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pull_request</span><span>:
</span><span style="background-color:#282828;color:#569cd6;">jobs</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">build</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref != 'refs/heads/main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">build_and_deploy</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runs-on</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ubuntu-latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">github.ref == 'refs/heads/main</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="color:#d69d85;">'Checkout'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">actions/checkout@main</span><span>
</span></code></pre>
<p>Here’s a template that you can use for your own push vs PR workflows. By default, I assume <code>main</code> as the special branch, so you’ll need to change that if you want to use a different branch. Additionally, you’ll need to provide all the steps to take after checking code out.</p>
<h3 id="sources">Sources<a class="zola-anchor" href="#sources" aria-label="Anchor link for: sources">🔗</a></h3>
<ul>
<li><a rel="noopener nofollow" target="_blank" href="https://github.community/t/how-to-trigger-an-action-on-push-or-pull-request-but-not-both/16662/3">Github Community question: How to trigger an action on push or pull request but not both?</a></li>
<li><a rel="noopener nofollow" target="_blank" href="https://help.github.com/en/actions/reference/events-that-trigger-workflows">Github Actions reference: Events that trigger workflows</a> </li>
<li><a rel="noopener nofollow" target="_blank" href="https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners">Github Actions reference: Virtual Environments for Github-hosted runners</a></li>
<li><a rel="noopener nofollow" target="_blank" href="https://help.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context">Github Actions reference: Context and expression syntax - Github context</a></li>
</ul>
How to link multiple docker-compose services via network2020-06-11T00:00:00+00:002020-06-11T00:00:00+00:00https://tjtelan.com/blog/how-to-link-multiple-docker-compose-via-network/<p>This scenario came from a question I was asked docker-compose and network connectivity between services defined in different docker-compose.yml files.</p>
<p>The desired result was to be able to define a docker-compose.yml in one file, and in a second docker-compose.yml have the ability to reach the first service via service or container name for development purposes.</p>
<h2 id="default-scenario-two-separate-docker-compose-yml-and-two-separate-default-networks">Default scenario: Two separate docker-compose.yml and two separate default networks<a class="zola-anchor" href="#default-scenario-two-separate-docker-compose-yml-and-two-separate-default-networks" aria-label="Anchor link for: default-scenario-two-separate-docker-compose-yml-and-two-separate-default-networks">🔗</a></h2>
<p>Let’s take a simple docker compose file.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#d69d85;">'3'
</span><span style="background-color:#282828;color:#569cd6;">services</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service1</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">busybox</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">command</span><span>: </span><span style="background-color:#282828;color:#d69d85;">sleep infinity</span><span>
</span></code></pre>
<p>When it starts up, a default network is created. Its name is based on the service name and the directory name of the docker-compose.yml file.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ pwd
</span><span>/tmp/docker-example/compose1
</span><span>
</span><span>$ docker-compose up -d
</span><span>Creating network </span><span style="color:#d69d85;">"compose1_default"</span><span> with the default driver
</span><span>Creating compose1_service1_1 ... done
</span></code></pre>
<h3 id="second-docker-compose-file">Second docker compose file<a class="zola-anchor" href="#second-docker-compose-file" aria-label="Anchor link for: second-docker-compose-file">🔗</a></h3>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#d69d85;">'3'
</span><span style="background-color:#282828;color:#569cd6;">services</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service2</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">busybox</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">command</span><span>: </span><span style="background-color:#282828;color:#d69d85;">sleep infinity</span><span>
</span></code></pre>
<p>Starting services in a second docker compose file, we see the same behavior. A new default network is created and used.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ pwd
</span><span>/tmp/docker-example/compose2
</span><span>
</span><span>$ docker-compose up -d
</span><span>Creating network </span><span style="color:#d69d85;">"compose2_default"</span><span> with the default driver
</span><span>Creating compose2_service2_1 ... done
</span></code></pre>
<p>A side-effect of these isolated networks are that the containers are unable to ping one another by service name or container name.</p>
<h3 id="test-from-service-1-ping-service-2">Test: From Service 1 ping Service 2<a class="zola-anchor" href="#test-from-service-1-ping-service-2" aria-label="Anchor link for: test-from-service-1-ping-service-2">🔗</a></h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># By service name
</span><span>$ docker exec -it compose1_service1_1 ping service2
</span><span>ping: bad address </span><span style="color:#d69d85;">'service2'
</span><span>
</span><span style="color:#608b4e;"># By container name
</span><span>$ docker exec -it compose1_service1_1 ping compose2_service2_1
</span><span>ping: bad address </span><span style="color:#d69d85;">'compose2_service2_1'
</span></code></pre>
<h3 id="test-service-2-ping-service-1">Test: Service 2 ping Service 1<a class="zola-anchor" href="#test-service-2-ping-service-1" aria-label="Anchor link for: test-service-2-ping-service-1">🔗</a></h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># By service name
</span><span>$ docker exec -it compose2_service2_1 ping service1
</span><span>ping: bad address </span><span style="color:#d69d85;">'service1'
</span><span>
</span><span style="color:#608b4e;"># By container name
</span><span>$ docker exec -it compose2_service2_1 ping compose1_service1_1
</span><span>ping: bad address </span><span style="color:#d69d85;">'compose1_service1_1'
</span></code></pre>
<h2 id="new-scenario-sharing-a-network-between-services">New scenario: Sharing a network between services<a class="zola-anchor" href="#new-scenario-sharing-a-network-between-services" aria-label="Anchor link for: new-scenario-sharing-a-network-between-services">🔗</a></h2>
<p>If you want define services in multiple docker-compose.yml files, and also have network connectivity between the services, you need to configure your services to use the same network.</p>
<p>To create an external network, you can run <code>docker network create <name></code>. -- where <code><name></code> can be a single string without spaces.</p>
<h3 id="creating-the-network">Creating the network<a class="zola-anchor" href="#creating-the-network" aria-label="Anchor link for: creating-the-network">🔗</a></h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ docker network create external-example
</span><span>2af4d92c2054e9deb86edaea8bb55ecb74f84a62aec7614c9f09fee386f248a6
</span></code></pre>
<h3 id="modified-first-docker-compose-file-with-network-configured">Modified first docker-compose file with network configured<a class="zola-anchor" href="#modified-first-docker-compose-file-with-network-configured" aria-label="Anchor link for: modified-first-docker-compose-file-with-network-configured">🔗</a></h3>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#d69d85;">'3'
</span><span style="background-color:#282828;color:#569cd6;">services</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service1</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">busybox</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">command</span><span>: </span><span style="background-color:#282828;color:#d69d85;">sleep infinity</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">networks</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">default</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">external</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">external-example</span><span>
</span></code></pre>
<p>Restarting the services</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ pwd
</span><span>/tmp/docker-example/compose1
</span><span>
</span><span>$ docker-compose up -d
</span><span>Creating compose1_service1_1 ... done
</span></code></pre>
<h3 id="modified-second-docker-compose-file-with-network-configured">Modified second docker-compose file with network configured<a class="zola-anchor" href="#modified-second-docker-compose-file-with-network-configured" aria-label="Anchor link for: modified-second-docker-compose-file-with-network-configured">🔗</a></h3>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#d69d85;">'3'
</span><span style="background-color:#282828;color:#569cd6;">services</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service2</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">busybox</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">command</span><span>: </span><span style="background-color:#282828;color:#d69d85;">sleep infinity</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">networks</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">default</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">external</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">external-example</span><span>
</span></code></pre>
<p>Restarting the services</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ pwd
</span><span>/tmp/docker-example/compose2
</span><span>
</span><span>$ docker-compose up -d
</span><span>Creating compose2_service2_1 ... done
</span></code></pre>
<p>After running <code>docker-compose up -d</code> on both docker-compose.yml files, we see that no new networks were created.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>$ docker network ls
</span><span>NETWORK ID NAME DRIVER SCOPE
</span><span>25e0c599d5e5 bridge bridge local
</span><span>2af4d92c2054 external-example bridge local
</span><span>7df4631e9cff host host local
</span><span>194d4156d7ab none null local
</span></code></pre>
<p>With the containers using the <code>external-example</code> network, they are able to ping one another.</p>
<h3 id="test-service-1-ping-service-2">Test: Service 1 ping Service 2<a class="zola-anchor" href="#test-service-1-ping-service-2" aria-label="Anchor link for: test-service-1-ping-service-2">🔗</a></h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># By service name
</span><span>$ docker exec -it compose1_service1_1 ping service2
</span><span>PING service2 (172.24.0.3): 56 data bytes
</span><span>64 bytes from 172.24.0.3: seq=0 ttl=64 time=0.054 ms
</span><span>^C
</span><span>--- service2 ping statistics ---
</span><span>1 packets transmitted, 1 packets received, 0% packet loss
</span><span>round-trip min/avg/max = 0.054/0.054/0.054 ms
</span><span>
</span><span style="color:#608b4e;"># By container name
</span><span>$ docker exec -it compose1_service1_1 ping compose2_service2_1
</span><span>PING compose2_service2_1 (172.24.0.2): 56 data bytes
</span><span>64 bytes from 172.24.0.2: seq=0 ttl=64 time=0.042 ms
</span><span>^C
</span><span>--- compose2_service2_1 ping statistics ---
</span><span>1 packets transmitted, 1 packets received, 0% packet loss
</span><span>round-trip min/avg/max = 0.042/0.042/0.042 ms
</span></code></pre>
<h3 id="test-service-2-ping-service-1-1">Test: Service 2 ping Service 1<a class="zola-anchor" href="#test-service-2-ping-service-1-1" aria-label="Anchor link for: test-service-2-ping-service-1-1">🔗</a></h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># By service name
</span><span>$ docker exec -it compose2_service2_1 ping service1
</span><span>PING service1 (172.24.0.2): 56 data bytes
</span><span>64 bytes from 172.24.0.2: seq=0 ttl=64 time=0.041 ms
</span><span>^C
</span><span>--- service1 ping statistics ---
</span><span>1 packets transmitted, 1 packets received, 0% packet loss
</span><span>round-trip min/avg/max = 0.041/0.041/0.041 ms
</span><span>
</span><span style="color:#608b4e;"># By container name
</span><span>$ docker exec -it compose2_service2_1 ping compose1_service1_1
</span><span>PING compose1_service1_1 (172.24.0.3): 56 data bytes
</span><span>64 bytes from 172.24.0.3: seq=0 ttl=64 time=0.042 ms
</span><span>^C
</span><span>--- compose1_service1_1 ping statistics ---
</span><span>1 packets transmitted, 1 packets received, 0% packet loss
</span><span>round-trip min/avg/max = 0.042/0.042/0.042 ms
</span></code></pre>
<blockquote>
<p>As a note, you can configure your services to use a custom container name by declaring the <code>container_name</code> key under each service (i.e., at the same level as <code>image</code>).</p>
<p><a rel="noopener nofollow" target="_blank" href="https://docs.docker.com/compose/compose-file/#container_name">Link to Docker-compose docs - container_name</a></p>
</blockquote>
<h2 id="takeaway">Takeaway<a class="zola-anchor" href="#takeaway" aria-label="Anchor link for: takeaway">🔗</a></h2>
<p>You can connect services defined across multiple docker-compose.yml files.</p>
<p>In order to do this you’ll need to:</p>
<ol>
<li>Create an external network with <code>docker network create <network name></code></li>
<li>In each of your docker-compose.yml configure the default network to use your externally created network with the <code>networks</code> top-level key.</li>
<li>You can use either the service name or container name to connect between containers.</li>
</ol>
Published First Crate on Crates.io2020-02-14T00:00:00+00:002020-02-14T00:00:00+00:00https://tjtelan.com/blog/published-first-crate-on-crates-io/<p>I published <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/git-url-parse">my first public crate</a>. I thought my library was useful, general, and did not have a similar implementation in crates.io. I hoped that it may get used by the Rust community. It turned out to be very easy to package and upload my code, and I wanted to share my process.</p>
<h2 id="complete-cargo-toml-with-package-metadata">Complete Cargo.toml with package metadata<a class="zola-anchor" href="#complete-cargo-toml-with-package-metadata" aria-label="Anchor link for: complete-cargo-toml-with-package-metadata">🔗</a></h2>
<p>https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata</p>
<p>I tried to define the metadata completely. Other than writing the code, this took the longest time. I wanted to make sure the crate listed in an appropriate category. As well as using relevant keywords.</p>
<p>(My very first version did not have a README or fancy build badges, but I did have docstrings for docs.rs)</p>
<h2 id="generate-api-key-and-log-in-from-cargo">Generate API Key and log in from cargo<a class="zola-anchor" href="#generate-api-key-and-log-in-from-cargo" aria-label="Anchor link for: generate-api-key-and-log-in-from-cargo">🔗</a></h2>
<p>Crates.io only supports logging in using Github accounts.</p>
<p>Navigate to Account Settings and scroll down to the <em>API Access</em> section. Click new token and give your token a name.</p>
<p>After giving your token a name, there is going to be a <code>cargo login</code> command with a random token value. Run this command to log in.</p>
<h2 id="run-cargo-publish">Run cargo publish<a class="zola-anchor" href="#run-cargo-publish" aria-label="Anchor link for: run-cargo-publish">🔗</a></h2>
<p>My crate was at the top of the new crates column of crates.io</p>
<p>Like I mentioned earlier, I had doc strings in my code that I expected to publish to docs.rs. This can take a few minutes. Wait a few minutes. It’ll make it there.</p>
<p>I later followed up with writing a complete README.md, and added badges. That’s all it took to make my little library look fancy.</p>
<p>If you were on the fence about publishing to crates.io, I hope you are now convinced that it was not.</p>
<hr />
<h4 id="before-you-go-some-info-about-my-crate">Before you go, some info about my crate:<a class="zola-anchor" href="#before-you-go-some-info-about-my-crate" aria-label="Anchor link for: before-you-go-some-info-about-my-crate">🔗</a></h4>
<p><a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/git-url-parse">git-url-parse</a>.</p>
<p>I wrote it because common ssh-based git repo urls don’t fit a standard like:</p>
<ul>
<li><a rel="noopener nofollow" target="_blank" href="https://url.spec.whatwg.org/">the living URL standard</a></li>
<li><a rel="noopener nofollow" target="_blank" href="https://tools.ietf.org/html/rfc1738">RFC 1738</a></li>
<li><a rel="noopener nofollow" target="_blank" href="https://tools.ietf.org/html/rfc1808.html">RFC 1808</a></li>
<li>etc.</li>
</ul>
<p>As such, parsing is not supported by many languages’ standard library, including Rust.</p>
<p>Anyway, I hope you'll check it out! Thanks!</p>
Using a database + gRPC with Rust2019-04-25T00:00:00+00:002020-12-03T00:00:00+00:00https://tjtelan.com/blog/using-a-database-grpc-with-rust/<blockquote>
<p>Update: 8/19/2020</p>
<p>If you're here for <a rel="noopener nofollow" target="_blank" href="https://grpc.io/">gRPC</a> or CLI, this post uses old crates and you should read <a href="https://tjtelan.com/blog/lets-build-a-single-binary-grpc-server-client-with-rust-in-2020/">this post</a> instead.</p>
<p>I have a new post using more up-to-date Rust with <code>async</code>/<code>await</code>. The new guide uses <a rel="noopener nofollow" target="_blank" href="https://github.com/hyperium/tonic">Tonic</a> for gRPC and <a rel="noopener nofollow" target="_blank" href="https://github.com/TeXitoi/structopt">StructOpt</a> for CLI. (But no database stuff)</p>
<p>This post is still good if you want to see how to use <a rel="noopener nofollow" target="_blank" href="http://diesel.rs/">Diesel</a> with <a rel="noopener nofollow" target="_blank" href="https://www.postgresql.org/">PostgreSQL</a>. But if you're here for gRPC, you should check out the <a href="https://tjtelan.com/blog/lets-build-a-single-binary-grpc-server-client-with-rust-in-2020/">new post</a>.</p>
<p>Thanks for listening! Now back to your regularly scheduled program...</p>
</blockquote>
<hr />
<p>This is a summary of my experience with writing a Rust DB-backed server/client with <a rel="noopener nofollow" target="_blank" href="https://github.com/pingcap/grpc-rs">grpc-rs</a> to communicate to the backend, and <a rel="noopener nofollow" target="_blank" href="http://diesel.rs/">Diesel</a> as an ORM to be used with <a rel="noopener nofollow" target="_blank" href="https://www.postgresql.org/">PostgreSQL</a>.</p>
<h2 id="what-did-i-want-out-of-this-exercise">What did I want out of this exercise?<a class="zola-anchor" href="#what-did-i-want-out-of-this-exercise" aria-label="Anchor link for: what-did-i-want-out-of-this-exercise">🔗</a></h2>
<p>I don't consider myself an expert with Rust, also not a beginner. I've been following the Rust language development for a while. I also have been wanting to move from writing code for personal projects to writing for work projects. </p>
<p>I have the privilege to choose the tools I want at work, but I must keep in mind that I don't work by myself. I need to be able to provide practical development advice and enough technical mentorship to my teammates to keep us all productive.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://medium.com/@KevinHoffman/streaming-grpc-with-rust-d978fece5ef6">Kevin Hoffman’s blog post</a> let me know that what I wanted was possible today in stable (as opposed to nightly). Kevin’s post is great, but I couldn’t really absorb it my first few reads, because he is a more experienced Rust developer than myself. I didn't quite understand the code in his post, and I couldn’t appreciate details he skimmed over which I will point out. I hope that I can provide supplemental details.</p>
<h3 id="my-target">My target<a class="zola-anchor" href="#my-target" aria-label="Anchor link for: my-target">🔗</a></h3>
<p>I am looking to build a very basic command line interface client, and a backend service. The cli communicates to the backend via gRPC, and the backend connects to a database.</p>
<p><strong>gRPC</strong></p>
<p>Based on Kevin Hoffman's experience, and the download activity on crates.io, I also used Pingcap's library <a rel="noopener nofollow" target="_blank" href="https://github.com/pingcap/grpc-rs">grpc-rs</a>. However, while writing this post <a rel="noopener nofollow" target="_blank" href="https://github.com/tower-rs/tower-grpc">tower-rs</a> (which is a pure Rust implementation) is considered to be stable, though may not yet implement all features.</p>
<p><strong>Database</strong></p>
<p>For database, I decided to use <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/diesel">Diesel-rs</a> since there really aren’t any other choices that I felt were better in a production environment. Diesel is a mature project that is very actively supported.</p>
<p><strong>Command line interface</strong></p>
<p>For the command line interface, I picked <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/clap">clap-rs</a>, because I was interested in trying out defining the command line content and structure with yaml. In the future I would probably use <a rel="noopener nofollow" target="_blank" href="https://crates.io/crates/structopt">StructOpt</a>. It happens to use clap-rs internally, but the written code is easier for me to read, and in my opinion, less code to write derives. For this reason, I’ll probably gloss over the command line implementation. It provides the minimal amount of interaction I needed to highlight what appears to be an idiomatic pattern. </p>
<p>After spending a few hours with all the tools, I wanted to jump in feet first with an example project.</p>
<h2 id="my-first-attempt-figuring-out-my-development-pattern">My first attempt figuring out my development pattern<a class="zola-anchor" href="#my-first-attempt-figuring-out-my-development-pattern" aria-label="Anchor link for: my-first-attempt-figuring-out-my-development-pattern">🔗</a></h2>
<p>I briefly considered not telling the parts of the story where I was figuring out how to get everything to compile but here it is. It ended up being a big learning experience. I won't get into super deep detail about my intentions since I ended up not going in this direction. But I will highlight what I learned.</p>
<p>I focused on individually building with Diesel and gRPC. Once I felt ready to do something productive with these crates, I started thinking about implementation by designing the protocol buffers first, and designing the database later. This ended up being a time-expensive mistake that hopefully will not need to repeated, dear Reader.</p>
<h3 id="red-flags-in-the-workflow">Red flags in the workflow<a class="zola-anchor" href="#red-flags-in-the-workflow" aria-label="Anchor link for: red-flags-in-the-workflow">🔗</a></h3>
<p>I am generating my proto Rust code from <code>.proto</code> using <a rel="noopener nofollow" target="_blank" href="https://github.com/pingcap/grpc-rs">grpc-rs</a> in my <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/blob/main/cli-clap-grpc-pingcap-db-diesel/workspace/protos/build.rs">build.rs</a>. It runs during <code>cargo build</code>. Based on Diesel's <a rel="noopener nofollow" target="_blank" href="http://diesel.rs/guides/getting-started/">getting started</a> guide, I expected that I would be annotating my proto Rust with the same <code>#[derive()]</code>. </p>
<p>But If I'm going to be using the generated structs w/ Diesel, then I have to break up the protobuf compilation w/ some manual step to additionally add in the correct annotations, because the next <code>cargo build</code> regenerated code and removed my manual changes. This was a red flag, but I kept moving forward anyway...</p>
<p>Diesel also expects that your struct fields are 1:1 with your table schema for to use the custom <code>#[Derive(Queryable)]</code> for querying the DB. If you haven't looked at <code>grpc-rs</code> generated grpc code, you'll see extra internally used struct fields: <code>unknown_fields</code> and <code>cached_size</code>. These are part of <code>grpc-rs</code>’s implementation of message serialization/deserialization. Moving forward could require representing these extra fields in the database, which has a bad smell and is wasteful of space. </p>
<p><strong>Example of grpc-rs generated Rust code w/ the special fields</strong></p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[derive(PartialEq,Clone,Default)]
</span><span style="color:#569cd6;">pub struct </span><span>OrderForm {
</span><span> </span><span style="color:#608b4e;">// message fields
</span><span> </span><span style="color:#569cd6;">pub </span><span>quantity: </span><span style="color:#569cd6;">i32</span><span>,
</span><span> </span><span style="color:#569cd6;">pub </span><span>product: OilProductType,
</span><span> </span><span style="color:#608b4e;">// special fields
</span><span> </span><span style="color:#569cd6;">pub </span><span>unknown_fields: ::protobuf::UnknownFields,
</span><span> </span><span style="color:#569cd6;">pub </span><span>cached_size: ::protobuf::CachedSize,
</span><span>}
</span></code></pre>
<blockquote>
<p>Choosing to work directly with this generated struct means manually modifying the list of derive() and working around the special fields <code>unknown_fields</code> and <code>cached_size</code> so Diesel could still be used inserts and queries. Possibly requiring adding columns in the table schema. This is a more tight coupling than I want between my protobuf library and the data in the database.</p>
</blockquote>
<h3 id="what-i-should-have-done">What I should have done<a class="zola-anchor" href="#what-i-should-have-done" aria-label="Anchor link for: what-i-should-have-done">🔗</a></h3>
<p>I only realized this after writing the client/server using the raw proto structs. I then moved onto designing the db schema and migrations. I got stuck trying to flow the grpc client calls to db inserts.</p>
<p>I concluded that I would need to create new structs that only Diesel would use since their support heavily relies on Derive code. It all felt like an impedance mismatch, and I was having to redo the same work over again without a clear path for where I was going.</p>
<p>This was a failure. If I could work backwards from the database inserts to the protos, then this might work out better for my understanding.</p>
<h2 id="my-second-approach">My second approach<a class="zola-anchor" href="#my-second-approach" aria-label="Anchor link for: my-second-approach">🔗</a></h2>
<h3 id="before-implementation">Before implementation<a class="zola-anchor" href="#before-implementation" aria-label="Anchor link for: before-implementation">🔗</a></h3>
<p>I'm still learning how to write idiomatic Rust. When I got my protos compiling into generated Rust code, and assumed I needed to use it directly because it is native code, despite my unfamiliarity with all of the code generated by Pingcap’s gRPC library.</p>
<blockquote>
<p>I’m relying heavily on the use of the Into trait to create a little anti-corruption layer so that the business logic on both my client and my server are not operating directly on the protobuf-generated structs. <em>-- Kevin Hoffman</em></p>
</blockquote>
<p>After a not-skimmed reading of <a rel="noopener nofollow" target="_blank" href="https://medium.com/@KevinHoffman/streaming-grpc-with-rust-d978fece5ef6">Kevin's Hoffman's post</a>, I noticed he described using this same approach in a hand-wavey manner. I wasn't ready to appreciate the warning without some example code or a diagram.</p>
<h4 id="use-separate-structs-for-business-logic">Use separate structs for business logic<a class="zola-anchor" href="#use-separate-structs-for-business-logic" aria-label="Anchor link for: use-separate-structs-for-business-logic">🔗</a></h4>
<p>I hadn’t immediately considered that I might want to write my own structs instead of using the protobuf-generated structs since my mindset was that the generated code would be ergonomic enough to use code.</p>
<p>However, the strategy of using separate structs offers very easy to use conversions because of the <code>From</code> and <code>Into</code> traits. This would be easier for the maintainability and readability of my code because I can contain that conversion logic in away from my business logic.</p>
<p>I could convert them back and forth between the protobuf-generated forms and the diesel supported forms with <code>.into()</code>. How is this achieved?</p>
<p>More on this during implementation...</p>
<h5 id="what-is-using-this-pattern-like-in-the-code">What is using this pattern like in the code?<a class="zola-anchor" href="#what-is-using-this-pattern-like-in-the-code" aria-label="Anchor link for: what-is-using-this-pattern-like-in-the-code">🔗</a></h5>
<p>An example interaction would look like this</p>
<p>Inserts into the database - Client side:</p>
<ol>
<li>User input </li>
<li>Create Diesel struct + any data manipulation </li>
<li>Convert Diesel struct into Proto struct </li>
<li>Send Proto struct in gRPC call</li>
</ol>
<p>Inserts into the database - Server side:</p>
<ol>
<li>Receive Proto struct</li>
<li>Convert Proto struct into Diesel struct + Any data manipulation</li>
<li>Insert into DB</li>
</ol>
<h4 id="the-last-complicated-detail-rust-custom-types-mapping-to-postgres-enums">The Last complicated detail : Rust custom types mapping to Postgres Enums<a class="zola-anchor" href="#the-last-complicated-detail-rust-custom-types-mapping-to-postgres-enums" aria-label="Anchor link for: the-last-complicated-detail-rust-custom-types-mapping-to-postgres-enums">🔗</a></h4>
<p>I want to use Rust enums and Postgres enums to carry my usage of types all the way to DB insert/query. The diesel schema generator doesn't handle custom postgres enums well, but we can manage the conversion by hand by using a few Diesel Derives: <code>SqlType</code>, <code>FromSql</code>, and <code>ToSql</code> . I might cover using custom postgres types with Diesel in another post. But for now, I am going to hand-wave this detail.</p>
<p>The <a rel="noopener nofollow" target="_blank" href="https://github.com/diesel-rs/diesel/blob/v1.3.1/diesel_tests/tests/custom_types.rs">Diesel-rs custom types tests</a> were very useful helping me figure it out.</p>
<h4 id="organizing-code-into-cargo-workspaces">Organizing code into cargo workspaces<a class="zola-anchor" href="#organizing-code-into-cargo-workspaces" aria-label="Anchor link for: organizing-code-into-cargo-workspaces">🔗</a></h4>
<p>With some experience under my belt and a better understanding of where relative domains in the code should be separated by crate, I wanted to organize before writing new code. The first thing I did was separate the codebase into <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html#creating-a-workspace">workspaces</a>.</p>
<p>Separating into different crates would let me organize the struct conversion code from complicating the readability of the business logic code. This will make it easier to reuse patterns between the client and server side through importing the crates.</p>
<h3 id="implementation">Implementation<a class="zola-anchor" href="#implementation" aria-label="Anchor link for: implementation">🔗</a></h3>
<h4 id="write-database-schema">Write database schema<a class="zola-anchor" href="#write-database-schema" aria-label="Anchor link for: write-database-schema">🔗</a></h4>
<p>Because I need some kind of story to write code against, I decided to write an oil ordering system (because proto-diesel can be described as oil… har har…)</p>
<p>My postgres type <code>oil_product</code> has a <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Oil_refinery#/media/File:Usesofpetroleum.png">pie chart</a> of oil derived products that I got from the <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Oil_refinery#Major_products">wiki page of Oil Refinery</a></p>
<p>That helped me with my first thing: I need my database schema - <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/blob/main/cli-clap-grpc-pingcap-db-diesel/workspace/models/schema.rs">schema.rs</a></p>
<p>Then I could write my migrations:</p>
<ul>
<li><a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/blob/main/cli-clap-grpc-pingcap-db-diesel/workspace/migrations/2019-03-18-213310_create_orders/up.sql">up.sql</a></li>
<li><a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/blob/main/cli-clap-grpc-pingcap-db-diesel/workspace/migrations/2019-03-18-213310_create_orders/down.sql">down.sql</a></li>
</ul>
<h4 id="get-inserts-into-db-working">Get inserts into DB working<a class="zola-anchor" href="#get-inserts-into-db-working" aria-label="Anchor link for: get-inserts-into-db-working">🔗</a></h4>
<p>Second is getting inserts into the db working on the backend - <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/commit/0e40e27529170b22f5419559ce8659f7a1a154f3#diff-149a61a7aa6246849298372d0b2f196e">Link to specific commit</a></p>
<p><strong>backend.rs</strong></p>
<p>This is a simple call from the backend to an internal function that performs the DB insert. After opening a connection, I test create a hardcoded order.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span>[</span><span style="color:#569cd6;">...</span><span>]
</span><span style="color:#569cd6;">let</span><span> conn = client::establish_connection();
</span><span style="color:#569cd6;">let</span><span> new_order = client::create_order(</span><span style="color:#569cd6;">&</span><span>conn, </span><span style="color:#b5cea8;">1</span><span>, schema::OilProductEnum::</span><span style="color:#b4cea8;">DIESEL</span><span>);
</span><span>[</span><span style="color:#569cd6;">...</span><span>]
</span></code></pre>
<p><strong>create_order</strong></p>
<p>This insert only works once because the id is set to <code>1</code>. But the result is in insert of an order into the database, and returning the inserted <code>Order</code> from the function. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub fn </span><span>create_order(conn : </span><span style="color:#569cd6;">&</span><span>PgConnection, quantity : </span><span style="color:#569cd6;">i32</span><span>, product_type : OilProductEnum) -> Order {
</span><span> </span><span style="color:#569cd6;">let</span><span> new_order = vec![
</span><span> Order {
</span><span> id : </span><span style="color:#b5cea8;">1</span><span>,
</span><span> quantity : quantity,
</span><span> product_type : product_type,
</span><span> },
</span><span> ];
</span><span>
</span><span> diesel::insert_into(orders::table)
</span><span> .values(</span><span style="color:#569cd6;">&</span><span>new_order)
</span><span> .get_result(conn)
</span><span> .expect(</span><span style="color:#d69d85;">"Error saving new order"</span><span>)
</span><span>}
</span></code></pre>
<h5 id="creating-user-input-structs-for-business-logic">Creating user input structs for business logic<a class="zola-anchor" href="#creating-user-input-structs-for-business-logic" aria-label="Anchor link for: creating-user-input-structs-for-business-logic">🔗</a></h5>
<p>I created some structs solely for taking user input. It will converted to a proto form that will be used for gRPC calls</p>
<p>These structs didn't include dynamic info like ids or timestamps, since those are generated right before insert on the server side.</p>
<p>Separate proto messages needed to be defined specifically for taking user input from the client-side.</p>
<p><strong>One of the business logic structs</strong></p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub struct </span><span>OrderForm {
</span><span> </span><span style="color:#569cd6;">pub </span><span>quantity : </span><span style="color:#569cd6;">i32</span><span>,
</span><span> </span><span style="color:#569cd6;">pub </span><span>product_type : OilProductEnum,
</span><span>}
</span></code></pre>
<p><strong>The corresponding proto message definition</strong></p>
<pre data-lang="protobuf" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-protobuf "><code class="language-protobuf" data-lang="protobuf"><span>message OrderForm {
</span><span> int32 quantity = 2;
</span><span> OilProductType product = 3;
</span><span>}
</span></code></pre>
<h5 id="converting-business-logic-struct-to-from-proto-generated-struct">Converting business logic struct to/from proto-generated struct<a class="zola-anchor" href="#converting-business-logic-struct-to-from-proto-generated-struct" aria-label="Anchor link for: converting-business-logic-struct-to-from-proto-generated-struct">🔗</a></h5>
<p>I implemented the <code>From</code> trait to convert my custom type to protobuf types for the grpc client calls (and vice-versa). The <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/std/convert/trait.From.html">From</a> trait gives us the <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/std/convert/trait.Into.html">Into</a> implementation for free.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// Convert from the protos to our type
</span><span style="color:#569cd6;">impl </span><span>From<refinery::OrderForm> </span><span style="color:#569cd6;">for </span><span>OrderForm {
</span><span> </span><span style="color:#569cd6;">fn </span><span>from(proto_form : refinery::OrderForm) -> </span><span style="color:#569cd6;">Self </span><span>{
</span><span> OrderForm {
</span><span> quantity : proto_form.get_quantity(),
</span><span> product_type : OilProductEnum::from(proto_form.get_product()),
</span><span> }
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// Convert from our type to the proto
</span><span style="color:#569cd6;">impl </span><span>From<OrderForm> </span><span style="color:#569cd6;">for </span><span>refinery::OrderForm {
</span><span> </span><span style="color:#569cd6;">fn </span><span>from(rust_form : OrderForm) -> </span><span style="color:#569cd6;">Self </span><span>{
</span><span> </span><span style="color:#569cd6;">let mut</span><span> order = refinery::OrderForm::new();
</span><span>
</span><span> order.set_quantity(rust_form.quantity);
</span><span> order.set_product(refinery::OilProductType::from(rust_form.product_type));
</span><span> order
</span><span> }
</span><span>}
</span></code></pre>
<p>Snippet from <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/blob/main/cli-clap-grpc-pingcap-db-diesel/workspace/models/convert.rs">convert.rs</a>:</p>
<h4 id="take-user-input-before-making-grpc-call">Take user input before making gRPC call<a class="zola-anchor" href="#take-user-input-before-making-grpc-call" aria-label="Anchor link for: take-user-input-before-making-grpc-call">🔗</a></h4>
<p>We want to take user input from a client, make a grpc backend call, then insert into the db from the backend.</p>
<p>We already have taken care of converting to and from proto forms, so this is focused on control flow now.</p>
<h5 id="move-inserts-into-grpc-server-endpoint">Move inserts into gRPC server endpoint<a class="zola-anchor" href="#move-inserts-into-grpc-server-endpoint" aria-label="Anchor link for: move-inserts-into-grpc-server-endpoint">🔗</a></h5>
<p>Insert will occur after calling into the grpc server endpoint from the client-side.</p>
<p>On the client-side, I created a protobuf-generated struct with default values, for demonstrating the gRPC call to the backend works. I can easily take user input afterwards.</p>
<h5 id="receive-proto-struct-and-convert-into-db-insertable-struct">Receive proto struct and convert into DB insertable struct<a class="zola-anchor" href="#receive-proto-struct-and-convert-into-db-insertable-struct" aria-label="Anchor link for: receive-proto-struct-and-convert-into-db-insertable-struct">🔗</a></h5>
<p>Lastly, I worked out taking in user input, and using it to instantiate one of my custom types. During the grpc backend call, I call .into() on my type, which will convert to the protobuf form. On the backend, I take in the request, and call <code>.into()</code> to convert back into my type so I can marshal into a diesel insert call.</p>
<p><strong>Server-side</strong></p>
<p>I'm converting the proto-form struct <code>req</code> into the business logic form <code>OrderForm</code> by calling <code>.into()</code>. Since the <code>create_order()</code> impl takes in <code>OrderForm</code>, there is no need to annotate the type with <code>.into()</code> and we’re able to stay focused.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[derive(Clone)]
</span><span style="color:#569cd6;">struct </span><span>RefineryService;
</span><span>
</span><span style="color:#569cd6;">impl </span><span>Refinery </span><span style="color:#569cd6;">for </span><span>RefineryService {
</span><span> </span><span style="color:#608b4e;">// The client-side converts to refinery::OrderForm while calling this endpoint.
</span><span> </span><span style="color:#608b4e;">// But we convert the proto type back to our custom type right before adding to the database
</span><span> </span><span style="color:#569cd6;">fn </span><span>order(</span><span style="color:#569cd6;">&mut </span><span>self, ctx: RpcContext, req: refinery::OrderForm, sink: UnarySink<refinery::OrderStatus>) {
</span><span>
</span><span> </span><span style="color:#608b4e;">// Creating the return object
</span><span> </span><span style="color:#569cd6;">let</span><span> order_status = client::order_received_success();
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> f = sink
</span><span> .success(order_status.clone())
</span><span> .map(</span><span style="color:#569cd6;">move |_| </span><span>println!(</span><span style="color:#d69d85;">"Responded with status </span><span style="color:#e3bbab;">{{ </span><span style="color:#b4cea8;">{:?} </span><span style="color:#e3bbab;">}}</span><span style="color:#d69d85;">"</span><span>, order_status))
</span><span> .map_err(</span><span style="color:#569cd6;">move |</span><span>err</span><span style="color:#569cd6;">| </span><span>eprintln!(</span><span style="color:#d69d85;">"Failed to reply: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, err));
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> conn = client::establish_connection();
</span><span> </span><span style="color:#608b4e;">// Convert the received proto request into our native type
</span><span> </span><span style="color:#569cd6;">let</span><span> _new_order = client::create_order(</span><span style="color:#569cd6;">&</span><span>conn, req.into());
</span><span>
</span><span> ctx.spawn(f)
</span><span> }
</span><span>}
</span></code></pre>
<p><strong>Function for creating order</strong></p>
<p>We take the business logic form <code>order_form</code> and use it to create the insertable struct <code>new_order</code> with all of the column values for Diesel to execute.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">pub fn </span><span>create_order(conn : </span><span style="color:#569cd6;">&</span><span>PgConnection, order_form : OrderForm) -> Order {
</span><span> </span><span style="color:#569cd6;">let</span><span> timestamp = NaiveDateTime::from_timestamp(Utc::now().timestamp(),</span><span style="color:#b5cea8;">0</span><span>);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> new_order = vec![
</span><span> NewOrder {
</span><span> quantity : order_form.quantity,
</span><span> product_type : order_form.product_type,
</span><span> received_time : timestamp,
</span><span> },
</span><span> ];
</span><span>
</span><span> diesel::insert_into(orders::table)
</span><span> .values(</span><span style="color:#569cd6;">&</span><span>new_order)
</span><span> .get_result(conn)
</span><span> .expect(</span><span style="color:#d69d85;">"Error saving new order"</span><span>)
</span><span>}
</span></code></pre>
<h5 id="do-it-again-in-reverse-for-queries">Do it again, in reverse, for queries<a class="zola-anchor" href="#do-it-again-in-reverse-for-queries" aria-label="Anchor link for: do-it-again-in-reverse-for-queries">🔗</a></h5>
<p>Last task to cover is repeating all of this work, but for making queries.</p>
<p>This ended up being slightly off pattern from implementing <code>From</code> traits, because I am returning a list of Orders, and the From trait apparently is not easily implemented for a Vec to the protobuf Rust equivilent. If I were planning on shipping this code somewhere other than for demonstration, I probably would spend more time implementing <code>From</code>. I ended up getting lazy, and wrapped the manual conversion in a function that loops and uses my already implemented From traits on the <code>Order</code> type. </p>
<p><strong>user input side</strong></p>
<p>This client subcommand from the cli requests all of the orders from the database, then prints out the protobuf form as a demonstration. The next step would be converting the protobuf list into a Vec of some non-protobuf generated type.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">if let </span><span>Some(_matches) = matches.subcommand_matches(</span><span style="color:#d69d85;">"summary"</span><span>) {
</span><span> </span><span style="color:#569cd6;">let</span><span> empty_payload = protos::empty::Empty::new();
</span><span>
</span><span> </span><span style="color:#608b4e;">// Send the gRPC message
</span><span> </span><span style="color:#569cd6;">let</span><span> orders = client.get_all_records(</span><span style="color:#569cd6;">&</span><span>empty_payload).expect(</span><span style="color:#d69d85;">"RPC Failed!"</span><span>);
</span><span>
</span><span> </span><span style="color:#608b4e;">// Print all records from database
</span><span> println!(</span><span style="color:#d69d85;">"Order status: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, orders);
</span><span>}
</span></code></pre>
<p><strong>server endpoint</strong></p>
<p>The server takes in an empty proto type, so we don’t have to do any type conversions. We then call a function <code>client::get_all_orders()</code> that calls Diesel to return all the data in a table. Then we make another function call <code>client::db_query_to_proto()</code> to convert our native data into a gRPC sendable form.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>get_all_records(</span><span style="color:#569cd6;">&mut </span><span>self, ctx: RpcContext, _req: protos::empty::Empty, sink: UnarySink<refinery::OrderRecordList>){
</span><span> println!(</span><span style="color:#d69d85;">"Received request for all of the order records"</span><span>);
</span><span> </span><span style="color:#569cd6;">let</span><span> conn = client::establish_connection();
</span><span>
</span><span> </span><span style="color:#608b4e;">// Call out to db
</span><span> </span><span style="color:#569cd6;">let</span><span> query_results = client::get_all_orders(</span><span style="color:#569cd6;">&</span><span>conn);
</span><span>
</span><span> </span><span style="color:#608b4e;">// This conversion pattern is different than the plain `From` traits, because we
</span><span> </span><span style="color:#608b4e;">// have to handle the outer vector in a special way, but I want to be lazy
</span><span> </span><span style="color:#569cd6;">let</span><span> parsed_query_proto = client::db_query_to_proto(query_results);
</span><span> </span><span style="color:#608b4e;">//println!("Got results from the database: {:?}", query_results);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> f = sink
</span><span> .success(parsed_query_proto.clone())
</span><span> .map(</span><span style="color:#569cd6;">move |_| </span><span>println!(</span><span style="color:#d69d85;">"Responded with list of records </span><span style="color:#e3bbab;">{{ </span><span style="color:#b4cea8;">{:?} </span><span style="color:#e3bbab;">}}</span><span style="color:#d69d85;">"</span><span>, parsed_query_proto))
</span><span> .map_err(</span><span style="color:#569cd6;">move |</span><span>err</span><span style="color:#569cd6;">| </span><span>eprintln!(</span><span style="color:#d69d85;">"Failed to reply: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, err));
</span><span>
</span><span> ctx.spawn(f)
</span><span>}
</span></code></pre>
<p><strong>database query</strong></p>
<p>This function queries for everything in the orders table. There's nothing interesting here because Diesel handles everything. I just needed to annotate the type of vector that Diesel was going to return.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// get_all_orders is used by the backend
</span><span style="color:#569cd6;">pub fn </span><span>get_all_orders(conn : </span><span style="color:#569cd6;">&</span><span>PgConnection) -> Vec<Order> {
</span><span> </span><span style="color:#569cd6;">let</span><span> query : Vec<Order> = orders::table.select(orders::all_columns)
</span><span> .order_by(orders::id)
</span><span> .load(conn)
</span><span> .expect(</span><span style="color:#d69d85;">"Error getting all order records"</span><span>);
</span><span> query
</span><span>}
</span></code></pre>
<p><strong>query to protobuf list</strong></p>
<p>You don't need to use all of the Rust features all up front or not use Rust at all. We can all hopefully appreciate that this can still be understood.</p>
<p>I briefly tried to implement <code>From</code> for <code>Vec<Order></code>, but it became evident that it was going to take a little more effort than I was willing to spend at this moment. I'm first to admit that this is a bit of a hack, but that's fine for demonstration purposes. </p>
<p>Protobuf's <code>repeated</code> keyword in the Rust code has its own type like <code>Vec<T></code> called <code>RepeatedField</code> and we are simply looping through and creating a <code>Vec<refinery::OrderRecord></code> so we could use the conversion impl <code>from_vec</code>. The rest is for building the return data.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">// db_query_to_proto is used by the backend to convert a Vector of Order (from a Diesel select
</span><span style="color:#608b4e;">// query) into the proto native OrderRecordList. Implementing `From` for a Vector would have taken
</span><span style="color:#608b4e;">// longer, and used a wrapper type. That very well may be the more maintainable approach, but this
</span><span style="color:#608b4e;">// was quicker…
</span><span>
</span><span style="color:#569cd6;">pub fn </span><span>db_query_to_proto(rust_record : Vec<Order>) -> refinery::OrderRecordList {
</span><span> </span><span style="color:#569cd6;">let mut</span><span> proto_vec : Vec<refinery::OrderRecord> = Vec::new();
</span><span>
</span><span> </span><span style="color:#608b4e;">// Let's take advantage of the `From` trait
</span><span> </span><span style="color:#569cd6;">for</span><span> r </span><span style="color:#569cd6;">in</span><span> rust_record {
</span><span> proto_vec.push(refinery::OrderRecord::from(r));
</span><span> }
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> proto_order = protobuf::RepeatedField::from_vec(proto_vec);
</span><span>
</span><span> </span><span style="color:#569cd6;">let mut</span><span> proto_final = refinery::OrderRecordList::new();
</span><span> proto_final.set_order(proto_order);
</span><span> proto_final
</span><span>}
</span></code></pre>
<h2 id="in-conclusion">In conclusion<a class="zola-anchor" href="#in-conclusion" aria-label="Anchor link for: in-conclusion">🔗</a></h2>
<p>Rust library support for gRPC is here. ORM support with Diesel-rs has been here for a while. But if you want to use gRPC and Diesel in the same project, maybe you can learn from my experience and be productive.</p>
<p>Do not primarily use the protobuf generated Rust types throughout your codebase. Especially if you plan on using Diesel-rs to deal with database inserts/queries, because structs need to be 1:1 with your table schema for the smoothest experience using Diesel-rs.</p>
<p>Implement the <code>From</code>/<code>Into</code> traits to more effectively convert between business logic structs and your protobuf generated Rust structs.</p>
<p>Convert to the proto Rust types only to send/return data over gRPC calls and then immediately convert back to your business logic type on the receiving end.</p>
<p>You don't need to write perfect Rust code in one go.</p>
<hr />
<p>The code used throughout this post is located <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/rust-examples/tree/main/cli-clap-grpc-pingcap-db-diesel">here</a>.</p>
Deploy Postfix Gmail relay with Ansible on Raspberry Pi2018-11-27T00:00:00+00:002018-11-27T00:00:00+00:00https://tjtelan.com/blog/deploy-postfix-gmail-relay-with-ansible/<h2 id="why-would-we-want-to-do-this">Why would we want to do this?<a class="zola-anchor" href="#why-would-we-want-to-do-this" aria-label="Anchor link for: why-would-we-want-to-do-this">🔗</a></h2>
<p>The virtualization servers at work are running VMWare ESXi, with Vcenter Server Applicance (VCSA) as our bridge to using cool, free tools like Packer, and Terraform to automate my interactions with virtual resources.</p>
<p>A downside we discovered is VCSA's lack of support for SMTP that requires auth, which Google requires when you send mail through them.</p>
<p>Postfix can handle the anonymous request from VCSA, and send it out to gmail with provided creds.</p>
<h2 id="how-do-i-get-started">How do I get started?<a class="zola-anchor" href="#how-do-i-get-started" aria-label="Anchor link for: how-do-i-get-started">🔗</a></h2>
<p>Since we wanted to get an email whenever there as an issue with the virtualization servers, it made sense to hostthis service on its own hardware.</p>
<p>I am going to be hosting this service using a Raspberry Pi 3 model B running Raspbian Stretch, and configuring it from my host using Ansible. This detail is not critical for following this guide. Any Debian-derived OS (like Ubuntu) that Ansible supports will work for hosting.</p>
<p>You just need to make sure SSH is turned on, and that you have the IP address. (The default username/pass on RasPis is <code>pi</code>/<code>raspberry</code>)</p>
<p>At minimum, you need the following tools installed on your host:</p>
<ul>
<li>Python 3</li>
<li>Ansible 2.7</li>
</ul>
<p>Download this helpful role for installing Postfix. At the time of this writing, it was the best public Postfix Ansible role, because its documentation had examples of how to configure the deployment as a gmail relay. Very straight forward.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://github.com/Oefenweb/ansible-postfix">https://github.com/Oefenweb/ansible-postfix</a></p>
<p>If you install this role in your Ansible client's <code>role_path</code>, then you can use the example playbook I slightly modified, (and annotated) from the ansible-postfix README.</p>
<h3 id="example-ansible-playbook">Example ansible playbook<a class="zola-anchor" href="#example-ansible-playbook" aria-label="Anchor link for: example-ansible-playbook">🔗</a></h3>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span>---
</span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Setup basic raspberry pi host as SMTP relay (Rasbian)</span><span>
</span><span style="background-color:#282828;color:#569cd6;">hosts</span><span>:
</span><span> </span><span style="background-color:#282828;color:#d69d85;">mailproxy</span><span>
</span><span style="background-color:#282828;color:#569cd6;">vars</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_mynetworks</span><span>:
</span><span> </span><span style="color:#608b4e;"># This is the IPv4 localhost loopback subnet
</span><span> - </span><span style="color:#d69d85;">'127.0.0.0/8'
</span><span> </span><span style="color:#608b4e;"># This is the IPv4 mapped IPv6 localhost loopback subnet
</span><span> - </span><span style="color:#d69d85;">'[::ffff:127.0.0.0]/104'
</span><span> </span><span style="color:#608b4e;"># This is the IPv6 localhost loopback address
</span><span> - </span><span style="color:#d69d85;">'[::1]/128'
</span><span> </span><span style="color:#608b4e;"># This is the local private network subnet, like the IPv4 address space from your home router
</span><span> </span><span style="color:#608b4e;"># This addition allows other hosts on the network to send mail through this relay!
</span><span> - </span><span style="color:#d69d85;">'192.168.0.0/24'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_smtpd_relay_restrictions</span><span>:
</span><span> </span><span style="color:#608b4e;"># This says to permit requests if the client is in the $mynetworks whitelist
</span><span> </span><span style="color:#608b4e;"># http://www.postfix.org/postconf.5.html#permit_mynetworks
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">permit_mynetworks</span><span>
</span><span> </span><span style="color:#608b4e;"># This says relay the request if client is authenticated to the smtp server
</span><span> </span><span style="color:#608b4e;"># http://www.postfix.org/postconf.5.html#permit_sasl_authenticated
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">permit_sasl_authenticated</span><span>
</span><span> </span><span style="color:#608b4e;"># This says to reject the request unless it knows about the destination (the domain)
</span><span> </span><span style="color:#608b4e;"># http://www.postfix.org/postconf.5.html#reject_unauth_destination
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">reject_unauth_destination</span><span>
</span><span>
</span><span> </span><span style="color:#608b4e;">## Lastly, I believe the order of these restrictions matter, so this last one must catch the rest of the garbage requests
</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_relayhost</span><span>: </span><span style="background-color:#282828;color:#d69d85;">smtp.gmail.com</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_smtp_tls_cafile</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/etc/ssl/certs/ca-certificates.crt</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_relaytls</span><span>: </span><span style="color:#569cd6;">true
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_sasl_user</span><span>: </span><span style="color:#d69d85;">'username@gmail.com'
</span><span> </span><span style="background-color:#282828;color:#569cd6;">postfix_sasl_password</span><span>: </span><span style="color:#d69d85;">'apppasswordgeneratedgarbage'
</span><span style="background-color:#282828;color:#569cd6;">roles</span><span>:
</span><span> </span><span style="background-color:#282828;color:#d69d85;">ansible-postfix</span><span>
</span></code></pre>
<h4 id="some-additional-notes">Some additional notes<a class="zola-anchor" href="#some-additional-notes" aria-label="Anchor link for: some-additional-notes">🔗</a></h4>
<ul>
<li>To configure vCenter, I followed this guide. It might be helpful to note that I only found these instructions to work with the Flash-based client, not the HTML5-based client. But it would be really great if the settings could be configured over the command line with VMWare's vSphere CLI tool, <a rel="noopener nofollow" target="_blank" href="https://github.com/vmware/govmomi/tree/master/govc">govc</a>
<ul>
<li><a rel="noopener nofollow" target="_blank" href="https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcenterhost.doc/GUID-467DA288-7844-48F5-BB44-99DE6F6160A4.html">VMWare Docs - Configure Mail Sender Settings - VSphere 6.5</a></li>
</ul>
</li>
<li>Without the <code>postfix_mynetworks</code> addition of my local network, I was unable to successfully see email alerts from VCSA being sent from Postfix</li>
<li>This also differs from the Oefenweb/ansible-postfix example, in that I am not setting any <code>postfix_aliases</code>, since it was my experience that it didn't ever work. Email was always from whoever was configured as <code>postfix_sasl_user</code></li>
</ul>
<h3 id="test-the-configuration">Test the configuration<a class="zola-anchor" href="#test-the-configuration" aria-label="Anchor link for: test-the-configuration">🔗</a></h3>
<p>Here is how to send a test email, from the Raspberry Pi, using <code>mail</code></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>pi@raspberrypi:~ $ echo </span><span style="color:#d69d85;">"Hello world, it's ya boi, RaspberryPi" </span><span style="color:#569cd6;">| </span><span>mail -s </span><span style="color:#d69d85;">"[SMTP proxy] Hello World"</span><span> your.email@domain.com
</span></code></pre>
Building a Unix-shell in Rust - Part 42018-01-21T00:00:00+00:002018-01-21T00:00:00+00:00https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-4/<p>This is the 4th post in a running series about writing a simple unix shell in the Rust language.
I suggest you catch up on the previous posts before reading ahead! </p>
<ul>
<li><a href="https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-1/">part 1</a></li>
<li><a href="https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-2/">part 2</a></li>
<li><a href="https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-3/">part 3</a></li>
</ul>
<hr />
<p>Back to evaluating the parsed command. This time we are going to be implementing built-in functions.</p>
<h3 id="shell-builtins">Shell builtins<a class="zola-anchor" href="#shell-builtins" aria-label="Anchor link for: shell-builtins">🔗</a></h3>
<p>Let’s quickly review how a shell works.
User is prompted for input. The input is tokenized (we are naively splitting on spaces). The first element of the tokenized input is the keyword, and the rest are the arguments. We execute the keyword with the arguments.</p>
<p>Our keywords correspond to either a shell function call (a builtin) or an external binary in your executable search path, which we will cover when we look to execute binaries in the next part. (In Bash, you can view this path by looking at the value of the environmental variable PATH. <code>$ echo ${PATH}</code>)</p>
<p>Builtin keywords are functions that are implemented in the shell codebase. Calls to builtin commands are just local function calls. </p>
<p>In Bash, usually you can view what commands are implemented as shell functions with <code>$ man builtins</code>. (And some platforms use external binaries for many common builtins, rather than rely on the shell implementation)</p>
<p>Some common builtins, which we will implement are:</p>
<ul>
<li>echo</li>
<li>history</li>
<li>cd</li>
<li>pwd</li>
</ul>
<h3 id="my-initial-strategy">My initial strategy<a class="zola-anchor" href="#my-initial-strategy" aria-label="Anchor link for: my-initial-strategy">🔗</a></h3>
<p>I’m going to keep my strategy simple. When I input a command, I want to run the builtin command. If my input is not a builtin, then let’s throw an error saying the command isn’t found. This will set us up for when we execute binaries,.</p>
<p>The first thing we want to do when we process the command is evaluate if it is a builtin. If it is, we want to pass arguments to the builtin function. </p>
<p>I’m scratching my head a little bit about how to represent the mapping of a keyword to a function in an idiomatic way.</p>
<p>I’ve found the <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/std/collections/struct.HashMap.html">HashMap</a> module, which is part of the standard collection library, but I’m looking to see if I can use something else that doesn’t require importing a library. I think what I want is an <code>enum</code> and I can pattern match to call builtin functions.</p>
<p>After a little bit of thought, I wondered if I could parse the string into the enum? My google-ing informs me that to accomplish this, I need to implement the <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/std/str/trait.FromStr.html">fromStr</a> trait. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">enum </span><span>Builtin {
</span><span> Echo,
</span><span> History,
</span><span> Cd,
</span><span> Pwd
</span><span>}
</span><span>
</span><span style="color:#569cd6;">impl </span><span>FromStr </span><span style="color:#569cd6;">for </span><span>Builtin {
</span><span> </span><span style="color:#569cd6;">type </span><span style="color:#4ec9b0;">Err </span><span>= ();
</span><span> </span><span style="color:#569cd6;">fn </span><span>from_str(s : </span><span style="color:#569cd6;">&str</span><span>) -> Result<</span><span style="color:#569cd6;">Self</span><span>, </span><span style="color:#569cd6;">Self::</span><span>Err> {
</span><span> </span><span style="color:#569cd6;">match</span><span> s {
</span><span> </span><span style="color:#d69d85;">"echo" </span><span style="color:#569cd6;">=> </span><span>Ok(Builtin::Echo),
</span><span> </span><span style="color:#d69d85;">"history" </span><span style="color:#569cd6;">=> </span><span>Ok(Builtin::History),
</span><span> </span><span style="color:#d69d85;">"cd" </span><span style="color:#569cd6;">=> </span><span>Ok(Builtin::Cd),
</span><span> </span><span style="color:#d69d85;">"pwd" </span><span style="color:#569cd6;">=> </span><span>Ok(Builtin::Pwd),
</span><span> </span><span style="color:#569cd6;">_ => </span><span>Err(()),
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>This is how I use the enum to call the function if it is a builtin</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>process_command(c : Command) -> </span><span style="color:#569cd6;">i32 </span><span>{
</span><span> </span><span style="color:#569cd6;">match </span><span>Builtin::from_str(</span><span style="color:#569cd6;">&</span><span>c.keyword) {
</span><span> Ok(Builtin::Echo) </span><span style="color:#569cd6;">=> </span><span>builtin_echo(</span><span style="color:#569cd6;">&</span><span>c.args),
</span><span> Ok(Builtin::History) </span><span style="color:#569cd6;">=> </span><span>builtin_history(</span><span style="color:#569cd6;">&</span><span>c.args),
</span><span> Ok(Builtin::Cd) </span><span style="color:#569cd6;">=> </span><span>builtin_cd(</span><span style="color:#569cd6;">&</span><span>c.args),
</span><span> Ok(Builtin::Pwd) </span><span style="color:#569cd6;">=> </span><span>builtin_pwd(</span><span style="color:#569cd6;">&</span><span>c.args),
</span><span> </span><span style="color:#569cd6;">_ => </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">: command not found"</span><span>, </span><span style="color:#569cd6;">&</span><span>c.keyword);
</span><span> </span><span style="color:#b5cea8;">1
</span><span> },
</span><span> }
</span><span>}
</span></code></pre>
<p>Here’s an example of one of the builtins. (I’m only going to show one with functionality, because I’m going to implement the rest later)
I chose to implement echo because it is very easy to verify. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>builtin_echo(args : </span><span style="color:#569cd6;">&</span><span>Vec<String>) -> </span><span style="color:#569cd6;">i32 </span><span>{
</span><span> println!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">"</span><span>, args.join(</span><span style="color:#d69d85;">" "</span><span>));
</span><span> </span><span style="color:#b5cea8;">0
</span><span>}
</span></code></pre>
<p>The number I'm returning signal that the command is done executing and represent the exit code of the command. 0 is conventionally a successful call, and anything else is an error.
And here we are in action:</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo run
</span><span> Finished debug </span><span style="color:#569cd6;">[</span><span>unoptimized + debuginfo</span><span style="color:#569cd6;">]</span><span> target(s) in 0.0 secs
</span><span> Running `target/debug/rust-shell`
</span><span>% echo test test test
</span><span>DEBUG: Raw input: </span><span style="color:#d69d85;">"echo test test test\n"
</span><span>DEBUG: Split input: </span><span style="color:#569cd6;">[</span><span style="color:#d69d85;">"echo"</span><span>, </span><span style="color:#d69d85;">"test"</span><span>, </span><span style="color:#d69d85;">"test"</span><span>, </span><span style="color:#d69d85;">"test"</span><span style="color:#569cd6;">]
</span><span>DEBUG: keyword : </span><span style="color:#d69d85;">"echo"
</span><span>DEBUG: args : </span><span style="color:#569cd6;">[</span><span style="color:#d69d85;">"test"</span><span>, </span><span style="color:#d69d85;">"test"</span><span>, </span><span style="color:#d69d85;">"test"</span><span style="color:#569cd6;">]
</span><span>test test test
</span><span>DEBUG: Exit code : 0
</span><span>% not_a_real_command lkfjdslf lkjfwe
</span><span>DEBUG: Raw input: </span><span style="color:#d69d85;">"not_a_real_command lkfjdslf lkjfwe\n"
</span><span>DEBUG: Split input: </span><span style="color:#569cd6;">[</span><span style="color:#d69d85;">"not_a_real_command"</span><span>, </span><span style="color:#d69d85;">"lkfjdslf"</span><span>, </span><span style="color:#d69d85;">"lkjfwe"</span><span style="color:#569cd6;">]
</span><span>DEBUG: keyword : </span><span style="color:#d69d85;">"not_a_real_command"
</span><span>DEBUG: args : </span><span style="color:#569cd6;">[</span><span style="color:#d69d85;">"lkfjdslf"</span><span>, </span><span style="color:#d69d85;">"lkjfwe"</span><span style="color:#569cd6;">]
</span><span>not_a_real_command: command not found
</span><span>DEBUG: Exit code : 1
</span></code></pre>
<p>I think I’m going to use this break to do some minor cleanup, write tests, and start using the rust logging mechanisms, such as the <a rel="noopener nofollow" target="_blank" href="https://github.com/rust-lang-nursery/log">log</a> crate. I’ll be back in the next post for running executables.</p>
Building a Unix-shell in Rust - Part 32017-12-31T00:00:00+00:002017-12-31T00:00:00+00:00https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-3/<p>This is the third post in a series on writing a simple shell in the Rust language. (I suggest you start from the <a href="https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-1/">beginning</a>!) </p>
<p>In the <a href="https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-2/">previous post</a> , I implemented a simple REPL that simply prints out debug output with the input split by whitespace.</p>
<hr />
<p>In this post, I would like to take the opportunity to set up tests before much more complex functionality gets included. Consider this to be the first part of what potentially might be multiple posts about writing and organizing testing with <code>cargo</code>.</p>
<h3 id="is-testing-important">Is testing important?<a class="zola-anchor" href="#is-testing-important" aria-label="Anchor link for: is-testing-important">🔗</a></h3>
<blockquote>
<p>Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.
<a rel="noopener nofollow" target="_blank" href="https://en.wikiquote.org/wiki/Edsger_W._Dijkstra">Edsger W. Dijkstra</a></p>
</blockquote>
<p>I don’t think it is controversial to say I think it is important. Good tests can help protect you from accidental regressions in functionality, and can be an added check on your assumptions, and manual testing. My intention is to write objective unit tests that will replace what I've been doing manually. </p>
<h3 id="why-write-the-tests-now-why-not-later">Why write the tests now? Why not later?<a class="zola-anchor" href="#why-write-the-tests-now-why-not-later" aria-label="Anchor link for: why-write-the-tests-now-why-not-later">🔗</a></h3>
<p>To be honest, I want the tests now because I’m looking for ways to use Rust for production code at work, and I need to get a feel for how a Rust codebase matures. As I am exploring Rust, I have come to be impressed with how easy <code>cargo test</code> makes it to write and execute tests.</p>
<p>It also will be less work to write test code for a small amount of code I just wrote now, rather than a larger amount of code later. Since it's been my experience that testing will just become reactive. And I get to go back to writing new feature code sooner.</p>
<h3 id="getting-started-with-unit-testing">Getting started with unit testing<a class="zola-anchor" href="#getting-started-with-unit-testing" aria-label="Anchor link for: getting-started-with-unit-testing">🔗</a></h3>
<p>Testing is something that I always seem to go through with print statements, which is better than nothing, but not the most reliable way to be mindful of functionality regression. I would like to try to write more tests, as well as more functional code.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/testing.html">https://doc.rust-lang.org/book/testing.html</a></p>
<p>According to the official Rust handbook, for unit-style tests, like the what I would like to write, the convention is to create a <code>tests</code> module.</p>
<p>I’ll cover integration tests in a later post, when I reorganize the project into different files. For now, I’m going to start slow and try to understand the new parts of Rust I get to use.</p>
<p>In the same file as the rest of my code, I add my test module with unit tests. I’m going to cover testing the <code>tokenize_command()</code> function.</p>
<p><strong>main.rs</strong></p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[cfg(test)]
</span><span style="color:#569cd6;">mod </span><span>unittest_tokenize_command {
</span><span> </span><span style="color:#569cd6;">use super</span><span>::*;
</span><span>
</span><span> #[test]
</span><span> #[ignore]
</span><span> </span><span style="color:#569cd6;">fn </span><span>empty_command() {
</span><span> assert_eq!(</span><span style="color:#d69d85;">""</span><span>, tokenize_command(</span><span style="color:#d69d85;">""</span><span>.to_string()).keyword)
</span><span> }
</span><span>
</span><span> #[test]
</span><span> </span><span style="color:#569cd6;">fn </span><span>test_keyword() {
</span><span> assert_eq!(</span><span style="color:#d69d85;">"test"</span><span>, tokenize_command(</span><span style="color:#d69d85;">"test"</span><span>.to_string()).keyword)
</span><span> }
</span><span>
</span><span> #[test]
</span><span> </span><span style="color:#569cd6;">fn </span><span>no_arg() {
</span><span> assert_eq!(</span><span style="color:#b5cea8;">0</span><span>, tokenize_command(</span><span style="color:#d69d85;">"test"</span><span>.to_string()).args.len())
</span><span> }
</span><span>
</span><span> #[test]
</span><span> </span><span style="color:#569cd6;">fn </span><span>one_arg() {
</span><span> assert_eq!(</span><span style="color:#b5cea8;">1</span><span>, tokenize_command(</span><span style="color:#d69d85;">"test one"</span><span>.to_string()).args.len())
</span><span> }
</span><span>
</span><span> #[test]
</span><span> </span><span style="color:#569cd6;">fn </span><span>multi_args() {
</span><span> assert_eq!(</span><span style="color:#b5cea8;">3</span><span>, tokenize_command(</span><span style="color:#d69d85;">"test one two three"</span><span>.to_string()).args.len())
</span><span> }
</span><span>
</span><span> #[test]
</span><span> #[ignore]
</span><span> </span><span style="color:#569cd6;">fn </span><span>quotes() {
</span><span> assert_eq!(</span><span style="color:#b5cea8;">2</span><span>, tokenize_command(</span><span style="color:#d69d85;">"test \”one two\” three"</span><span>.to_string()).args.len())
</span><span> }
</span><span>}
</span></code></pre>
<h3 id="breakdown-of-test-module">Breakdown of test module<a class="zola-anchor" href="#breakdown-of-test-module" aria-label="Anchor link for: breakdown-of-test-module">🔗</a></h3>
<p>I’ll introduce the new syntax.</p>
<h4 id="use-super">use super::*<a class="zola-anchor" href="#use-super" aria-label="Anchor link for: use-super">🔗</a></h4>
<p>The use of <code>use</code> is new to me in Rust. I assume it means I am bringing in the namespace scope from outside to the top-level (instead of using <code>super::</code> at every function call) </p>
<p>Since the test module is an inner module, we need to bring the functions from the outside scope into the module’s local scope. We can do this individually, but we can just use <code>*</code> to pull them all in, even though I’m not going to be testing them all right now.</p>
<p>For more information about this usage, look at the <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/crates-and-modules.html#re-exporting-with-pub-use">Crates and Modules</a> page in the Rust documentation.</p>
<h4 id="attributes">Attributes<a class="zola-anchor" href="#attributes" aria-label="Anchor link for: attributes">🔗</a></h4>
<p>The <code>#</code> lines are called <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/attributes.html">attributes</a>. Attributes are defined by the compiler, and are used for different things. As of Rust 1.17, we currently we cannot create our own attributes. I’ll quickly describe the attributes we use, (but here’s the <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/reference/attributes.html">reference</a> to all of the attributes.</p>
<h5 id="test">#[test]<a class="zola-anchor" href="#test" aria-label="Anchor link for: test">🔗</a></h5>
<p>The <code>#[test]</code> attribute labels the functions as tests to the rust compiler. This is how <code>cargo test</code> knows what functions to run for tests.</p>
<h5 id="ignore">#[ignore]<a class="zola-anchor" href="#ignore" aria-label="Anchor link for: ignore">🔗</a></h5>
<p>The <code>#[ignore]</code> attribute tells cargo to skip the test. (However, you can tell cargo to run the ignored tests by running <code>cargo test -- --ignored</code>) I am using this attribute, because as I started writing tests, I realized I hadn’t covered the functionality that would let the tests pass. I don’t want to forget to do this, so I’ll write the test now.</p>
<h5 id="cfg-test">#[cfg(test)]<a class="zola-anchor" href="#cfg-test" aria-label="Anchor link for: cfg-test">🔗</a></h5>
<p>In <code>#[cfg(test)]</code>, we’re using the <code>cfg</code> attribute on the <code>unittest_tokenize_command</code> module. In our usage, the attribute tells the Rust compiler to compile the module only when we are compiling tests, like when we run <code>cargo test</code>.</p>
<h3 id="running-the-tests">Running the tests<a class="zola-anchor" href="#running-the-tests" aria-label="Anchor link for: running-the-tests">🔗</a></h3>
<p>We just need to run <code>cargo test</code>.</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo test
</span><span> Compiling rust-shell v0.1.0 (file:///Users/telant/src/rust-shell)
</span><span> Finished debug </span><span style="color:#569cd6;">[</span><span>unoptimized + debuginfo</span><span style="color:#569cd6;">]</span><span> target(s) in 0.45 secs
</span><span> Running target/debug/deps/rust_shell-cdb27ec22ae15a63
</span><span>
</span><span>running 6 tests
</span><span>test unittest_tokenize_command::empty_command ... ignored
</span><span>test unittest_tokenize_command::quotes ... ignored
</span><span>test unittest_tokenize_command::no_arg ... ok
</span><span>test unittest_tokenize_command::multi_args ... ok
</span><span>test unittest_tokenize_command::test_keyword ... ok
</span><span>test unittest_tokenize_command::one_arg ... ok
</span><span>
</span><span>test result: ok. 4 passed</span><span style="color:#569cd6;">; </span><span>0 failed</span><span style="color:#569cd6;">; </span><span>2 ignored</span><span style="color:#569cd6;">; </span><span>0 measured
</span></code></pre>
<p>And we see that the all but our ignored tests pass, which is good enough for now!</p>
<p>In the next post, I’ll be covering evaluating built-in keywords.</p>
Building a Unix-shell in Rust - Part 22017-11-26T00:00:00+00:002017-11-26T00:00:00+00:00https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-2/<p>This is the 2nd part of a series where I document writing a command shell in Rust. In the <a href="https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-1/">previous post</a> I reviewed what a shell is, and broke that down into stages I can use to organize my code.</p>
<h3 id="getting-user-input">Getting user input<a class="zola-anchor" href="#getting-user-input" aria-label="Anchor link for: getting-user-input">🔗</a></h3>
<p>First thing we need to do is create a project. Let’s use Cargo to create this for us.</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo new --bin rust-shell
</span></code></pre>
<p>For now, I'm going to assume we are only running interactively. So I'm just going to get a simple loop set up that asks for an input, and echoes it back to me.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>std::io;
</span><span>
</span><span style="color:#569cd6;">fn </span><span>main() {
</span><span> </span><span style="color:#569cd6;">loop </span><span>{
</span><span> </span><span style="color:#569cd6;">let mut</span><span> command = String::new();
</span><span> io::stdin().read_line(</span><span style="color:#569cd6;">&mut</span><span> command)
</span><span> .expect(</span><span style="color:#d69d85;">"Failed to read in command"</span><span>);
</span><span> println!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{0}</span><span style="color:#d69d85;">"</span><span>, command);
</span><span> }
</span><span>}
</span></code></pre>
<p>I’m using std::io to read input into the mutable command variable binding, then I println() to echo my input back to the screen.</p>
<p>Then we build:</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo build
</span><span> Compiling rust-shell v0.1.0 (file:///Users/telant/src/rust-shell)
</span><span> Finished dev </span><span style="color:#569cd6;">[</span><span>unoptimized + debuginfo</span><span style="color:#569cd6;">]</span><span> target(s) in 0.25 secs
</span></code></pre>
<p>No errors.</p>
<p>And testing it out:</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo run
</span><span> Finished dev </span><span style="color:#569cd6;">[</span><span>unoptimized + debuginfo</span><span style="color:#569cd6;">]</span><span> target(s) in 0.0 secs
</span><span> Running `target/debug/rust-shell`
</span><span>123
</span><span>123
</span><span>
</span><span>test
</span><span>test
</span></code></pre>
<p>Great. Looks like that was easy.</p>
<p>What we see here is me trying 2 commands: <code>123</code> and <code>test</code>. We see the command printed right back. (Printing a short prompt might make that more obvious… ) </p>
<h3 id="parsing-the-input-into-tokens">Parsing the input into tokens<a class="zola-anchor" href="#parsing-the-input-into-tokens" aria-label="Anchor link for: parsing-the-input-into-tokens">🔗</a></h3>
<p>Next step is to break the user input from a single continuous string into smaller pieces that we can evaluate. </p>
<p>I am just going to tokenize the string using spaces as delimiters (because it is easy. It is, however, not always accurate, but, Dear Reader, I already know this is not the final way I am going to implement tokens. Splitting on spaces is too greedy of an approach. Quoted arguments are usually evaluated to be a single argument, including spaces, for uses that include passing entire raw strings to other programs. Let's not get perfection distract us. I'll stay focused on getting something that works...)</p>
<p>I actually ran into a little bit of trouble getting this working on a single line, with the original variable because of the type checker.</p>
<p>This did not work:</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo build
</span><span> Compiling rust-shell v0.1.0 (file:///Users/telant/src/rust-shell)
</span><span>error[E0282]: unable to infer enough type information about `B`
</span><span> --> src/main.rs:11:41
</span><span> </span><span style="color:#569cd6;">|
</span><span>11 </span><span style="color:#569cd6;">| </span><span>println!(</span><span style="color:#d69d85;">"{:?}"</span><span>, command.split(</span><span style="color:#d69d85;">' '</span><span>).collect());
</span><span> | ^^^^^^^ cannot infer type for `B`
</span><span> |
</span><span> = note: type annotations or generic parameter binding required
</span></code></pre>
<p>I’m lazy, and I didn’t look into how to explicitly reference the type.</p>
<p>This did work. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">let</span><span> command_split : Vec<</span><span style="color:#569cd6;">&str</span><span>> = command.split(</span><span style="color:#d69d85;">' '</span><span>).collect();
</span><span>println!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command_split);
</span></code></pre>
<p>This is the relevant output</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>test test test
</span><span>[</span><span style="color:#d69d85;">"test"</span><span>, </span><span style="color:#d69d85;">"test"</span><span>, </span><span style="color:#d69d85;">"test\n"</span><span>]
</span></code></pre>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>blah blah </span><span style="color:#d69d85;">"string in quotes"
</span><span>[</span><span style="color:#d69d85;">"blah"</span><span>, </span><span style="color:#d69d85;">"blah"</span><span>, </span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\"</span><span style="color:#d69d85;">string"</span><span>, </span><span style="color:#d69d85;">"in"</span><span>, </span><span style="color:#d69d85;">"quotes</span><span style="color:#e3bbab;">\"</span><span style="color:#d69d85;">\n"</span><span>]
</span></code></pre>
<p>I’m going to have to learn how type inference works in Rust sooner or later, but I’m not going to deal with it now. String types in Rust are kind of confusing coming from Python where I don’t have to deal with types very often.</p>
<p>(This is a warning from the future. You should lightly understand the idiomatic difference between String and &str. You find this out the hard way when you get to refactoring… see you in the future)</p>
<p>I’m going to use this moment to make the interface a more obvious when the we are ready to take user input by printing a prompt character.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>std::io::{self,Write};
</span><span>
</span><span style="color:#569cd6;">fn </span><span>main() {
</span><span> </span><span style="color:#569cd6;">let</span><span> prompt_char = </span><span style="color:#d69d85;">"%"</span><span>;
</span><span> </span><span style="color:#569cd6;">loop </span><span>{
</span><span> print!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{0} </span><span style="color:#d69d85;">"</span><span>, prompt_char);
</span><span> io::stdout().flush().unwrap();
</span><span>
</span><span> </span><span style="color:#569cd6;">let mut</span><span> command = String::new();
</span><span> io::stdin().read_line(</span><span style="color:#569cd6;">&mut</span><span> command)
</span><span> .expect(</span><span style="color:#d69d85;">"Failed to read in command"</span><span>);
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> command_split : Vec<</span><span style="color:#569cd6;">&str</span><span>> = command.split(</span><span style="color:#d69d85;">' '</span><span>).collect();
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command_split);
</span><span> }
</span><span>}
</span></code></pre>
<p>I added <code>DEBUG:</code> to our debug statements. Also I had to include a new <code>use</code>, use the <code>print!</code> macro, and flush the buffer so it would print to the screen immediately.</p>
<p>I got this pattern from the Rust docs for <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/1.4.0/std/macro.print!.html">print!</a></p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cargo run
</span><span> Finished dev </span><span style="color:#569cd6;">[</span><span>unoptimized + debuginfo</span><span style="color:#569cd6;">]</span><span> target(s) in 0.0 secs
</span><span> Running `target/debug/rust-shell`
</span><span>% Feeling a little more shell-like now
</span><span>DEBUG: </span><span style="color:#d69d85;">"Feeling a little more shell-like now\n"
</span><span>DEBUG: </span><span style="color:#569cd6;">[</span><span style="color:#d69d85;">"Feeling"</span><span>, </span><span style="color:#d69d85;">"a"</span><span>, </span><span style="color:#d69d85;">"little"</span><span>, </span><span style="color:#d69d85;">"more"</span><span>, </span><span style="color:#d69d85;">"shell-like"</span><span>, </span><span style="color:#d69d85;">"now\n"</span><span style="color:#569cd6;">]
</span><span>%
</span></code></pre>
<h3 id="classifying-parsed-input">Classifying parsed input<a class="zola-anchor" href="#classifying-parsed-input" aria-label="Anchor link for: classifying-parsed-input">🔗</a></h3>
<p>Last thing I’m going to do is identify the keyword from the arguments, then I’ll do a little refactoring to help organize the new complexity. (I expect to do a little fighting with the borrow checker at this point.)</p>
<p>Getting the keyword is easy. I just need to pick off the first element of our tokenized command.</p>
<p>The arguments is a vector slice of everything but the first element of the command. Can I slice a vector as easily as slicing lists in Python? Yes.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>std::io::{self,Write};
</span><span>
</span><span style="color:#569cd6;">fn </span><span>main() {
</span><span> </span><span style="color:#569cd6;">let</span><span> prompt_char = </span><span style="color:#d69d85;">"%"</span><span>;
</span><span> </span><span style="color:#569cd6;">loop </span><span>{
</span><span> print!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{0} </span><span style="color:#d69d85;">"</span><span>, prompt_char);
</span><span> io::stdout().flush().unwrap();
</span><span>
</span><span> </span><span style="color:#569cd6;">let mut</span><span> command = String::new();
</span><span> io::stdin().read_line(</span><span style="color:#569cd6;">&mut</span><span> command)
</span><span> .expect(</span><span style="color:#d69d85;">"Failed to read in command"</span><span>);
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: Raw input: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> command_split : Vec<</span><span style="color:#569cd6;">&str</span><span>> = command.split(</span><span style="color:#d69d85;">' '</span><span>).collect();
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: Split input: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command_split);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> keyword = command_split[</span><span style="color:#b5cea8;">0</span><span>];
</span><span> </span><span style="color:#569cd6;">let</span><span> arguments = </span><span style="color:#569cd6;">&</span><span>command_split[</span><span style="color:#b5cea8;">1</span><span style="color:#569cd6;">..</span><span>];
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: Keyword: </span><span style="color:#b4cea8;">{0}</span><span style="color:#d69d85;">"</span><span>, keyword);
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: Number of arguments: </span><span style="color:#b4cea8;">{0:?}</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">DEBUG: Arguments: </span><span style="color:#b4cea8;">{1:?}</span><span style="color:#d69d85;">"</span><span>, arguments.len(), arguments);
</span><span> }
</span><span>}
</span></code></pre>
<p>I have to call the slice by reference using <code>&</code> with the vector, and I specified the range I wanted to slice with the element I want to start from and <code>..</code> without an ending element. Rust figures out the bounds in this case.</p>
<h3 id="time-to-refactor">Time to refactor!<a class="zola-anchor" href="#time-to-refactor" aria-label="Anchor link for: time-to-refactor">🔗</a></h3>
<p>I’m going to make the main loop look a little more functional (inside the loop).</p>
<p>Printing the prompt? Easy. Function call.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>print_prompt() {
</span><span> </span><span style="color:#569cd6;">let</span><span> prompt_char = </span><span style="color:#d69d85;">"%"</span><span>;
</span><span>
</span><span> print!(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">{0} </span><span style="color:#d69d85;">"</span><span>, prompt_char);
</span><span> io::stdout().flush().unwrap();
</span><span>}
</span></code></pre>
<p>Reading the command from user input? I had to look up how to return variables. The style is to use an implicit return, and no semicolon. You can use <code>return</code>, but it isn't very idiomatic. </p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">fn </span><span>read_command() -> String {
</span><span> </span><span style="color:#569cd6;">let mut</span><span> command = String::new();
</span><span> io::stdin().read_line(</span><span style="color:#569cd6;">&mut</span><span> command)
</span><span> .expect(</span><span style="color:#d69d85;">"Failed to read in command"</span><span>);
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: Raw input: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command);
</span><span>
</span><span> command
</span><span>}
</span></code></pre>
<p>Tokenizing the command? Gonna get a little more complicated. I’m going to set up a struct to represent the command so I can keep the tokenized command together in a single object.</p>
<hr />
<h3 id="optional-first-fight-with-borrow-checker">Optional : First fight with borrow checker<a class="zola-anchor" href="#optional-first-fight-with-borrow-checker" aria-label="Anchor link for: optional-first-fight-with-borrow-checker">🔗</a></h3>
<p>I almost lost the motivation to continue the documenting my thought process because of this obstacle. This section can be skipped if you are looking to follow my happy path, and don't want to follow my confusion. </p>
<p>(This is what I wrote first, when I was actually having a fight with the borrow checker…)</p>
<p>I have to learn a little bit about <a rel="noopener nofollow" target="_blank" href="https://doc.rust-lang.org/book/lifetimes.html">lifetimes</a> in order to get this to compile. This makes some sense, since the struct will need to own the slice data, and in the original code, we were just borrowing the slice.</p>
<p>I’m finding it confusing thinking about what I need to do in order to make the tokenizing function use the Command struct. If I can copy the args to the struct, and give ownership of the string to the struct, then I assume this will compile?</p>
<p>What type is the copied slice, and how do I specify that in the struct? How do I use the lifetime in code to find my use case? I don’t even know what other questions to ask next.</p>
<p>Rather than try to figure out how to compile, and get the struct working with tokenizing the command, I’ll try to play around in main() and try instantiating my struct.</p>
<p>What I need to be able to do is copy the data in the vector. I tried for a while trying to pass ownership of a slice, but I ended up finding a way to take the first element out of the vector, and having the rest be the arguments be what is left. I feel a little over my head at this point, and I’m going to spend some time reading the docs.</p>
<p>This is what the struct looked like.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#608b4e;">/// Bad.
</span><span style="color:#569cd6;">struct </span><span>Command <'a> {
</span><span> keyword : String,
</span><span> arguments : </span><span style="color:#569cd6;">&'a</span><span> [</span><span style="color:#569cd6;">&'a str</span><span>],
</span><span> }
</span></code></pre>
<p>The reason I went with this approach was I thought I could pass the ownership of the heap from when I split the command by whitespace. This was really not a good approach, and I wasted quite a lot of time fighting with the borrow checker.</p>
<hr />
<h3 id="back-to-the-show">Back to the show<a class="zola-anchor" href="#back-to-the-show" aria-label="Anchor link for: back-to-the-show">🔗</a></h3>
<p>I ended up changing the way I split the original command string so I would have a Vec<String> rather than Vec<&str>. Because String is owned and &str is borrowed, and the Command struct needs to own its data. </p>
<p>I think I have a much more straightforward function.</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">struct </span><span>Command {
</span><span> keyword : String,
</span><span> args : Vec<String>,
</span><span>}
</span><span>
</span><span style="color:#569cd6;">fn </span><span>tokenize_command(c : String) -> Command {
</span><span> </span><span style="color:#569cd6;">let mut</span><span> command_split : Vec<String> = c.split_whitespace().map(|s| s.to_string()).collect();
</span><span> println!(</span><span style="color:#d69d85;">"DEBUG: Split input: </span><span style="color:#b4cea8;">{:?}</span><span style="color:#d69d85;">"</span><span>, command_split);
</span><span>
</span><span> </span><span style="color:#569cd6;">let</span><span> command = Command {
</span><span> keyword : command_split.remove(</span><span style="color:#b5cea8;">0</span><span>),
</span><span> args : command_split,
</span><span> };
</span><span>
</span><span> command
</span><span>}
</span></code></pre>
<p>Before getting to the next step of evaluating the parsed command, I want to take a moment to learn how to set up tests that will run with the builds. See you next time. </p>
Building a Unix-shell in Rust - Part 12017-11-05T00:00:00+00:002017-11-05T00:00:00+00:00https://tjtelan.com/blog/building-a-unix-shell-in-rust-part-1/<p>My goal is to find more work opportunities to write in Rust the same way I can write in Python and Go. Since I spend a lot of time designing and executing automation, it felt useful to start somewhere familiar. How about a simple Unix shell? Yes, I use bash all the time.</p>
<p>Rather than get this all worked out before posting, I'm going to document as much of my thought process in the design, as I have it. (But I am editing this to spare you the noisier stream-of-consciousness experience.)</p>
<p>I'll have code snippets occasionally, but I'm trying to keep the audience around intermediate experience (where I consider myself to be today). I'm going to assume you use another programming language today to Get Shit Done, and use the terminal to do simple things, but not necessarily write shell scripts.</p>
<p>Why am I doing this? I don't often see posts from beginning Rust learners doing practical, simple things (that can simply be copy/pasted and modified slightly), like in the other more mature language communities... Widest market? Probably not. </p>
<p>I guess that's enough rambling. Let’s dive in.</p>
<hr />
<h2 id="what-s-a-shell">What’s a shell?<a class="zola-anchor" href="#what-s-a-shell" aria-label="Anchor link for: what-s-a-shell">🔗</a></h2>
<p>A shell is an interactive language interpreter that allows you to run text-based commands and translates them into an action, such as making internal function calls, or running external programs.</p>
<p>You usually use it to access resources from the operating system. </p>
<p>Additional to accepting a text command - it typically outputs text results and/or causes some other side-effect.</p>
<p>You may know some of the name brand shells like the kind we're making, a Unix shell:</p>
<ul>
<li>Bourne-shell (sh)</li>
<li>bash</li>
<li>zsh</li>
<li>fish</li>
</ul>
<p>Or the windows specific:</p>
<ul>
<li>Command Prompt (cmd.exe)</li>
<li>Powershell</li>
</ul>
<p>Or interpreted languages:</p>
<ul>
<li>python </li>
<li>lua</li>
<li>haskell </li>
</ul>
<p>Shells run in terminal emulators. This is (over-) simplified as the text-only window that runs your shell. </p>
<p>It handles the interaction from you (known as Standard-In, like keystrokes) and your shell (known as Standard-Out for the buffered/flushed output style, and Standard-Error for the direct output style).</p>
<p>In most cases, the terminal emulator and shell are different processes (Windows’ cmd.exe and Powershell are confusingly, both the shell and terminal emulator) </p>
<p>You may have made reference to it by other common names such as:</p>
<ul>
<li>command prompt</li>
<li>terminal</li>
<li>console</li>
</ul>
<p>Examples of some terminal emulators</p>
<ul>
<li>xterm</li>
<li>rxvt</li>
<li>iTerm</li>
<li>Terminal.app</li>
<li>Windows Command Prompt</li>
<li>Powershell</li>
</ul>
<p>I’m going to focus on writing a bash-like shell. Functionality, and syntax should feel familiar. </p>
<hr />
<p>The shells are a <a rel="noopener nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>, a Read-Eval-Print-Loop.</p>
<p>Typically, a character (let’s say ‘$’) is printed and a cursor blinks. This informs the user that a command can be typed in.</p>
<p>You type in a command.</p>
<p>You hit enter to translate the command into an action.</p>
<p>The output of the program prints to the screen.</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$
</span></code></pre>
<h3 id="what-does-this-mean-for-me">What does this mean for me?<a class="zola-anchor" href="#what-does-this-mean-for-me" aria-label="Anchor link for: what-does-this-mean-for-me">🔗</a></h3>
<p>It lets us quickly stub the code out into this REPL pattern
Our main function that will enter a loop. Inside the main loop, we request a command from the user, and do something. Then we do it all over again. </p>
<h2 id="breaking-down-the-steps">Breaking down the steps<a class="zola-anchor" href="#breaking-down-the-steps" aria-label="Anchor link for: breaking-down-the-steps">🔗</a></h2>
<h3 id="first-we-read">First we Read<a class="zola-anchor" href="#first-we-read" aria-label="Anchor link for: first-we-read">🔗</a></h3>
<p>We need to take user input. Most shells print a symbol to signal to the user that we can input a command (as opposed to, for example, executing a command). I need to learn how to get a text command from the user. </p>
<p>Let's start our definition of a command. </p>
<p><strong>Command</strong></p>
<blockquote>
<p>a series of words separated by spaces.</p>
</blockquote>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ keyword arg1 arg2 arg3…
</span></code></pre>
<p>The first word is a keyword. It's either a built-in function or an executable on the filesystem, with the rest of the line being parameters passed to our function. </p>
<p><strong>Keyword</strong></p>
<blockquote>
<p>One of 2 possibilities :</p>
<ul>
<li>A built-in function to the shell (that is, calling a function in the code)</li>
<li>An executable
<ul>
<li>Either in one of the directories in your PATH</li>
<li>Or a filesystem path (relative or absolute)</li>
</ul>
</li>
</ul>
</blockquote>
<h3 id="then-we-execute">Then we Execute<a class="zola-anchor" href="#then-we-execute" aria-label="Anchor link for: then-we-execute">🔗</a></h3>
<p>When we use a command that calls a built-in, we simply pass the arguments to the function, and return back to the start of the loop when it completes. </p>
<h4 id="and-when-we-call-an-executable">And when we call an executable?<a class="zola-anchor" href="#and-when-we-call-an-executable" aria-label="Anchor link for: and-when-we-call-an-executable">🔗</a></h4>
<p>We need to make a <a rel="noopener nofollow" target="_blank" href="https://en.m.wikipedia.org/wiki/Fork_(system_call)">fork</a> syscall, that is, create a new process for the executable to run in, so it can have its own memory space, and manage its own interactions with the operating system. (The shell is still the parent process) </p>
<p>To start a process inside the child process, we have to call the <a rel="noopener nofollow" target="_blank" href="https://en.m.wikipedia.org/wiki/Exec_(system_call)">exec</a> syscall. </p>
<h3 id="then-we-process">Then we Process<a class="zola-anchor" href="#then-we-process" aria-label="Anchor link for: then-we-process">🔗</a></h3>
<p>This is when we cause side-effects to the system.</p>
<p>We want to provide feedback to the user to let them know the results of this process. To keep this simple, we will only consider returning text to the user, as we are providing commands as text. </p>
<p>Our shell process has at least 3 file descriptors for passing input, or receiving output provided. Stdin, Stdout, and Stderr. I need to know how to do that purely with Rust. </p>
<p>After the process is complete, any output should be printed to the screen, via stdout or stderr.</p>
<p>Exit codes will be treated as binary for this exercise. It should be set to 0 if we exit without error. Otherwise the exit code will be 1.</p>
<h3 id="lastly-we-loop">Lastly, we Loop<a class="zola-anchor" href="#lastly-we-loop" aria-label="Anchor link for: lastly-we-loop">🔗</a></h3>
<p>Return of control will go back to the user. The default user prompt will print as a visual cue (along with the typical blinking cursor) and we should be able to enter another command. </p>
<h2 id="the-coding-strategy">The coding strategy<a class="zola-anchor" href="#the-coding-strategy" aria-label="Anchor link for: the-coding-strategy">🔗</a></h2>
<p>So then we're running commands. Let's review the strategy. </p>
<ul>
<li>I need to know how to take input command in a loop. </li>
<li>I need to process the input to separate the keyword from the arguments </li>
<li>I need a way to call both builtins and executables. </li>
<li>The most abstract : I need to give the user feedback about the command run. (E. g. Print onto screen as appropriate and set an exit code of the command.) </li>
</ul>
<p>In the next post, we'll dive into using <code>cargo</code> and start writing in Rust. </p>
Learning New Build Systems2017-04-01T00:00:00+00:002017-04-01T00:00:00+00:00https://tjtelan.com/blog/learning-new-build-systems/<p>I do a lot of devops automation for work. This gives me a lot of opportunity to touch the build systems of many different languages. I have a professional background with Python, and a lot of experience building C/C++ with make.</p>
<p>I most recently got to play with Java/Maven, and NodeJS/Gulp.</p>
<p>--</p>
<p>The project I was building has a UI written in node, and built into an npm module with gulp. There are also maven pom files that describe the build with gulp using npm run build. This was done so everything could be orchestrated for building into a java jar.</p>
<p>Things I thought were neat about maven is the plugin system.</p>
<p>I'm not a huge fan of the pom files using xml, but it is really nice to be able to configure dependencies, and dynamically define build artifacts. The local maven repo is still a little confusing at times, but with more time I think it'll make more sense.</p>
<p>--</p>
<p>The UI build with gulp was more unfamiliar. The build instructions are in javascript, naturally. You can use <code>gulp.task()</code> and define a list of dependencies. But I had some experiences where my dependency lists were not made explicit enough. I suppose this is not much different than makefiles.</p>
Status Update2016-06-25T00:00:00+00:002016-06-25T00:00:00+00:00https://tjtelan.com/blog/status-update/<p>Another Status update post.</p>
<p>Its been about 3 months since starting my new job. I get to work on a lot of devops-y type of tasks, which is fun, and a much more positive life change.</p>
<h2 id="quick-rundown-on-some-things-i-ve-completed-so-far-in-no-order-of-importance-possibly-future-topics">Quick rundown on some things I've completed so far, in no order of importance. Possibly future topics<a class="zola-anchor" href="#quick-rundown-on-some-things-i-ve-completed-so-far-in-no-order-of-importance-possibly-future-topics" aria-label="Anchor link for: quick-rundown-on-some-things-i-ve-completed-so-far-in-no-order-of-importance-possibly-future-topics">🔗</a></h2>
<ul>
<li>Learn to get used to using OS X as my primary development system</li>
<li>Get familiar with using Homebrew for installing packages</li>
<li>Play with Ansible in an OS X environment</li>
<li>Beginning to use Chef to manage the team's testing environment.</li>
<li>Learn about Fluentd, to process development logs (and run arbitrary commands based on events. Lots here to talk about)</li>
<li>A lot of Docker! Docker-machine, Docker-compose with Virtualbox, then native Docker engine (1.12) in OS X</li>
<li>Some integrating of Eclipse and Docker via Docker Tooling plugin. (Perhaps more to talk about in the future)</li>
<li>AWS. Just getting familiar with this environment. Mostly S3, EC2, ECR, CloudFront, so far...</li>
<li>Shippable for CI. (They use containers for building our github check-ins. I think this is an interesting service)</li>
<li>Blackbox frontend testing with Selenium and Appium</li>
</ul>
<h2 id="topics-outside-of-tech-i-ve-been-spending-time-doing">Topics outside of tech I've been spending time doing<a class="zola-anchor" href="#topics-outside-of-tech-i-ve-been-spending-time-doing" aria-label="Anchor link for: topics-outside-of-tech-i-ve-been-spending-time-doing">🔗</a></h2>
<ul>
<li>Working on the house (I don't have a functional workshop yet, but soon)</li>
<li>Time and goal management</li>
<li>Watching a lot of YouTube</li>
<li>Getting more involved with many social media type apps</li>
<li>Gardening</li>
<li>Biking</li>
</ul>
<p>It's always been a goal of mine to make this blog something to hold myself accountable. So even if it is a post once in a while, I want to get my voice out there. In an effort to post more often, I'll be experimenting with this space to post non-tech stuff that hopefully brings some value.</p>
Taking Notes2015-07-16T00:00:00+00:002015-07-16T00:00:00+00:00https://tjtelan.com/blog/taking-notes/<p>I've carrired a bound, grid paper <a rel="noopener nofollow" target="_blank" href="http://www.bookfactory.com/engineering-notebooks/engineering-notebooks.html">engineering notebook</a> for quite a few years now. (A habit I picked up in school from the Electrical Engineering side of my formal education.) I've written an entry for (almost) every day working.</p>
<p>There are a few reasons why this has been my preference to text-based note taking, for example: a simple text file, or more complex, a wiki</p>
<h3 id="low-tech-solution-for-a-low-tech-problem">Low-tech solution for a low-tech problem<a class="zola-anchor" href="#low-tech-solution-for-a-low-tech-problem" aria-label="Anchor link for: low-tech-solution-for-a-low-tech-problem">🔗</a></h3>
<p>If I need to write a to-do list, or a reminder for myself while I'm away from my desk, it is a lot easier to jot something down and be done. I don't require to be near a text-editor. I don't need to create a wiki page, or do any post-markup to what I've written.</p>
<p>Also, I'm free to use a page however I need to in order to get my thought onto the page. I'm not limited to software features for my visual metaphors if I'm sketching out a diagram, or using common symbols</p>
<h3 id="the-downsides">The downsides<a class="zola-anchor" href="#the-downsides" aria-label="Anchor link for: the-downsides">🔗</a></h3>
<p>This approach has bitten me a few times. When I've started in with a new notebook in the middle of a project (since bound books have limited pages) I'm stuck carrying 2 books for a while.</p>
<p>Text searching features don't exist in my books. At least not without scanning and OCR (Making the assumtion that my handwriting is neat enough to be read into a text form, which is an unrealistic assumption.)</p>
<h3 id="looking-for-a-balance">Looking for a balance<a class="zola-anchor" href="#looking-for-a-balance" aria-label="Anchor link for: looking-for-a-balance">🔗</a></h3>
<p>It got me thinking about what I needed to do if I wanted to go all digital with my note taking, but keep my workflow intact.</p>
<p>The feeling of using a softer tip on a phone screen requires too much of a change to how I naturally write. I end up concentrating more just to make sure I write legibly enough for myself. Recently, I bought a <a rel="noopener nofollow" target="_blank" href="http://www.amazon.com/Adonit-Android-Samsung-Windows-Tablets/dp/B00R33ZYCG/ref=sr_1_1?s=pc&ie=UTF8&qid=1437058896&sr=1-1">hard-tipped stylus</a> for my phone, since my experiences with other rubber tipped styli were underwhelming. An argument can be made about the price for a good styli. The good reviews start at about $20-30. A large commitment for an instrument that is only good on touch screens.</p>
<p>I use an Android phone, and, at least at the time of this writing, there aren't a lot of great hand writing tools. Most are marketed towards the digital artist. Lots of authoring features I don't need, and some that are missing for creating journal entries.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://play.google.com/store/apps/details?id=com.fiistudio.fiinote">Fiinote</a> so far has been my favorite. I get a good mix of hand-written options, hand-drawn, text-based. And I can create hyperlinks, and embed images if I choose to. It has only been a few days, but if the free version is this great, the obvious choice may be to support the developers by upgrading to the full version.</p>
<h3 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h3>
<p>I recommend writing your notes down. There have been many <a rel="noopener nofollow" target="_blank" href="http://www.scientificamerican.com/article/a-learning-secret-don-t-take-notes-with-a-laptop/">articles</a> and <a rel="noopener nofollow" target="_blank" href="http://www.psychologicalscience.org/index.php/news/were-only-human/ink-on-paper-some-notes-on-note-taking.html">studies</a> about the cognitive advantages of longhand vs keyboarding for note taking.</p>
<p>Hard-tipped styli are pretty great, and cost about as much as nicer ink pens (~$25-100+).</p>
<p>Good, free software for handwriting exists for handwriting. Great, paid software undoubtedly exists.</p>
<p>Here's to a successful experiment!</p>
Goodbye Wordpress2015-07-05T00:00:00+00:002015-07-05T00:00:00+00:00https://tjtelan.com/blog/goodbye-wordpress/<p>I am in the process of migrating away from my self-hosted Wordpress to the Github-hosted <a rel="noopener nofollow" target="_blank" href="https://pages.github.com/">Github Pages</a>, powered by <a rel="noopener nofollow" target="_blank" href="http://jekyllrb.com/">Jekyll</a>.</p>
<p>I was finding it difficult to motivate myself to write when my wordpress comments kept getting hit by bots. Please excuse the plain styling while I get the hang of using Jekyll.</p>
Pip and bottle.py on NearlyFreeSpeech2013-12-06T00:00:00+00:002013-12-06T00:00:00+00:00https://tjtelan.com/blog/pip-and-bottle-py-on-nearlyfreespeech/<p>The biggest reason I like using nearlyfreespeech is the ability to prepay my hosting costs. The biggest downside (compared to other hosts) is a lack of straight-forward flexibility for my choice of web development environment. I appreciate the security-focused approach in how they offer features, but I can't say that it is always comfortable as a casual user.</p>
<ul>
<li>NFS has lots of language support, but non-PHP web development is only supported through CGI)</li>
</ul>
<p>I've been looking to try out bottle.py (partly because of their routing and partly because Django is not well supported on NFS as of this writing). I did my best to look for some examples online, but came up with only bits and pieces of the solution.</p>
<p>Here is how the environment was set up on <a rel="noopener nofollow" target="_blank" href="http://www.nearlyfreespeech.net/">NearlyFreeSpeech.net</a>:</p>
<h3 id="install-pip">Install pip<a class="zola-anchor" href="#install-pip" aria-label="Anchor link for: install-pip">🔗</a></h3>
<p>I prefer using <code>pip</code> over <code>easy_install</code>, so here's how to install that:</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#608b4e;">## Create your local site-packages directory.
</span><span style="color:#608b4e;"># In NFS's environment, this will end up being /home/private/.local
</span><span>$ mkdir -p ~/.local/lib/python2.7/site-packages
</span><span style="color:#608b4e;">## Automatically add this location to your execution path at login. Just for convenience.
</span><span>$ echo </span><span style="color:#d69d85;">'PATH=~/.local/bin:$PATH' </span><span>>> ~/.profile
</span><span style="color:#608b4e;">## Reload your .profile
</span><span>$ source ~/.profile
</span><span style="color:#608b4e;">## Use easy_install to install pip
</span><span>$ easy_install --prefix=~/.local pip
</span><span style="color:#608b4e;">## Now we can use pip to install bottle.py
</span><span>$ pip install --user bottle
</span></code></pre>
<p>This installed bottle.py in /home/private/.local/bin. Just for example
purposes, I copied this into my site-root so I wouldn't have to play with
python's sys.path. <em>Use your own judgement</em>.</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>$ cp ~/.local/bin/bottle.py /home/public
</span></code></pre>
<p>At this point, I was able to find a relevant <a rel="noopener nofollow" target="_blank" href="http://stackoverflow.com/questions/2664350/problems-with-routing-urls-using-cgi-and-bottle-py">stackoverflow question</a> specifically dealing with bottle.py and cgi.</p>
<p>The code is mostly unchanged (I added the shebang at the top), but I'll copy it here for copypasta purposes. Put bottle script and .htaccess in place in your site-root.</p>
<h4 id="home-public-index-py">/home/public/index.py<a class="zola-anchor" href="#home-public-index-py" aria-label="Anchor link for: home-public-index-py">🔗</a></h4>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;">#!/usr/local/bin/python
</span><span> </span><span style="color:#9b9b9b;">import </span><span>bottle
</span><span> </span><span style="color:#9b9b9b;">from </span><span>bottle </span><span style="color:#9b9b9b;">import </span><span>route
</span><span>
</span><span>@route(</span><span style="color:#d69d85;">'/'</span><span>)
</span><span> </span><span style="color:#569cd6;">def </span><span>index():
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#d69d85;">'Index'
</span><span>
</span><span>@route(</span><span style="color:#d69d85;">'/hello'</span><span>)
</span><span> </span><span style="color:#569cd6;">def </span><span>hello():
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#d69d85;">'Hello'
</span><span>
</span><span style="color:#569cd6;">if </span><span>__name__ == </span><span style="color:#d69d85;">'__main__'</span><span>:
</span><span> </span><span style="color:#9b9b9b;">from </span><span>wsgiref.handlers </span><span style="color:#9b9b9b;">import </span><span>CGIHandler
</span><span> CGIHandler().run(bottle.default_app())
</span></code></pre>
<h4 id="home-public-htaccess">/home/public/.htaccess<a class="zola-anchor" href="#home-public-htaccess" aria-label="Anchor link for: home-public-htaccess">🔗</a></h4>
<pre data-lang="cfg" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-cfg "><code class="language-cfg" data-lang="cfg"><span>DirectoryIndex index</span><span style="color:#569cd6;">.</span><span>py
</span><span>RewriteEngine </span><span style="color:#569cd6;">on
</span><span>RewriteCond %{REQUEST_FILENAME} </span><span style="color:#569cd6;">!-</span><span>f
</span><span>RewriteRule </span><span style="color:#d69d85;">^(.*)$ /index.py/$</span><span style="color:#b5cea8;">1</span><span> [</span><span style="color:#b5cea8;">L</span><span>]
</span></code></pre>
<p>Test this out by going to http://<domain>/ and http://<domain>/hello to confirm that bottle routes work. This works for me, but it is a little bit slow.</p>
<p>I'm excited to finally have my own non-php stuff to do on NFS. Hopefully you'll have fun with this as well.
Good luck.</p>
Learning Android to Play with Metawatch Manager2012-05-14T00:00:00+00:002012-05-14T00:00:00+00:00https://tjtelan.com/blog/learning-android-to-play-with-metawatch-manager/<p>I've been meaning to learn how to develop for Android since the <a rel="noopener nofollow" target="_blank" href="http://en.wikipedia.org/wiki/HTC_Dream">G1</a> (the first Android handset) was released.</p>
<p>I've also been meaning to develop for my Metawatch since I got it last September, but since launch, a lack of a <a rel="noopener nofollow" target="_blank" href="http://www.metawatch.org/forums/thread/200/official-free-cheap-development-environment">free/cheap development
environment</a> continues to slow my momentum (and interest) in messing with what is otherwise a nice looking, basic functioning watch. (That also has a motor in it...)</p>
<p>I'm going to take on both of these goals at the same time.</p>
<p>I was using the official version of the Metawatch Manager, the software installed in Android that is responsible for sending notifications (and text messages, phone calls...) to the watch to drive the motor and make you look at the watch.</p>
<p>I later moved onto an app I found in the Android market (err... Google Play). <a rel="noopener nofollow" target="_blank" href="https://play.google.com/store/apps/details?id=com.kupriyanov.metawatch&hl=en">Manager for MetaWatch</a> I've been finding that I really like what the app has offered me over the official one, but it is a little bit buggy and I'm also looking to add some features. It would be great to add the geeky, personal touch that I've been missing that shows that this is my project.</p>
<p>Anyway, I'm going to get back to the <a rel="noopener nofollow" target="_blank" href="http://lmgtfy.com/?q=android+tutorial">Android tutorial</a> I'm reading.</p>
<p><a rel="noopener nofollow" target="_blank" href="https://github.com/MetaWatchOpenProjects/MWM-for-Android">Official Metawatch Manager github project</a></p>
MetaWatch - Unboxing2011-09-16T00:00:00+00:002011-09-16T00:00:00+00:00https://tjtelan.com/blog/metawatch-initial-impression/<p>Without further delay, I'd like to post the first of several detailed posts about the new <a rel="noopener nofollow" target="_blank" href="http://www.metawatch.org">MetaWatch</a> that I received (after a <a rel="noopener nofollow" target="_blank" href="http://www.engadget.com/2011/07/11/fossil-wont-ship-the-meta-watch-until-august-dick-tracy-wannab/">couple</a> <a rel="noopener nofollow" target="_blank" href="http://www.engadget.com/2011/08/22/fossils-meta-watch-delayed-once-again-clearly-has-trouble-keep/">announcements</a> of delays) with a pretty brief unboxing post.</p>
<p>(I clearly already opened it immediately after I got it, but this is how it looked.)</p>
<p>[Unboxing pics]</p>
<p>The next post will be more focused on the watch itself. More pictures too.</p>
<p>Edit: Removed the links and the unboxing pics. Never got around to finishing the Metawatch first impression posts, but I still have it so I'll obviously be posting on any projects related to it.</p>
Laptop Volume Buttons in AwesomeWM2011-09-15T00:00:00+00:002011-09-15T00:00:00+00:00https://tjtelan.com/blog/laptop-volume-buttons-in-awesome-wm/<p>I've been using <a rel="noopener nofollow" target="_blank" href="http://www.archlinux.org">Arch Linux</a> on my laptop for a while now running <a rel="noopener nofollow" target="_blank" href="http://awesome.naquadah.org">Awesome</a> as my window manager.</p>
<p>I've have struggled to find the motivation to actually fix minor inconveniences because of the workarounds that I always seem to find first.</p>
<p>In this post, the inconvenience that I ended up solving is one that laptop users take for granted: the keyboard volume controls.</p>
<p>Initially, I just had trouble getting sound working. Turns out that I just
<a rel="noopener nofollow" target="_blank" href="https://wiki.archlinux.org/index.php/General_recommendations#Sound">needed to unmute</a>.</p>
<p>Just being overwhelmed by how much I need to read just to return some basic niceities back to my laptop usage, I've defaulted to the command line methods of doing what I need to do rather than read new documentation.</p>
<p>So that means that I've just been using <code>alsamixer</code> whenever I needed to control my volume since my volume keys did not have any function.</p>
<p>It turned out to be very simple. I just had to add a few lines to my awesome config file (<code>rc.lua</code>).</p>
<pre data-lang="lua" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-lua "><code class="language-lua" data-lang="lua"><span>awful.key({ }, </span><span style="color:#d69d85;">"XF86AudioMute"</span><span>, </span><span style="color:#569cd6;">function </span><span>() awful.util.spawn(</span><span style="color:#d69d85;">"amixer -c 0 set Master toggle"</span><span>) </span><span style="color:#569cd6;">end</span><span>),
</span><span>awful.key({ }, </span><span style="color:#d69d85;">"XF86AudioRaiseVolume"</span><span>, </span><span style="color:#569cd6;">function </span><span>() awful.util.spawn(</span><span style="color:#d69d85;">"amixer -c 0 set Master 2+ unmute"</span><span>) </span><span style="color:#569cd6;">end</span><span>),
</span><span>awful.key({ }, </span><span style="color:#d69d85;">"XF86AudioLowerVolume"</span><span>, </span><span style="color:#569cd6;">function </span><span>() awful.util.spawn(</span><span style="color:#d69d85;">"amixer -c 0 set Master 2-"</span><span>) </span><span style="color:#569cd6;">end</span><span>)
</span></code></pre>
<p>These went at the bottom of the globalkeys section in the rc.lua file. I'm not going to cover rc.lua, because if you don't know, then you really need to catch up on some documentation or you will probably screw things up.</p>
<p>I got the XF86* names by running <code>xev</code> and pressing the buttons. The amixer command <code>-c</code> flag is specific to my needs, so beware before copy/pasting.</p>
<p>When I get more comfortable with editing rc.lua, I plan on making a more detailed post.</p>
<p>Just taking a break between writing my review for my MetaWatch. Coming soon, I promise.</p>
Metawatch Has Arrived2011-09-12T00:00:00+00:002011-09-12T00:00:00+00:00https://tjtelan.com/blog/metawatch-has-arrived/<p>I just got my digital <a rel="noopener nofollow" target="_blank" href="http://www.metawatch.org">MetaWatch</a> this morning after 3 months of delays from TI.</p>
<p>I'm still playing around with the watch. Currently just got the Android apk installed on my phone and paired the watch to it. <del>There was no default UI to change the time</del>, <em>I was too lazy to read the user guide</em>, so I really needed to pair to use this as a watch.</p>
<p>There will be a more in-depth post soon, but here are a couple of pictures of the watch.</p>
<p><img src="https://lh6.googleusercontent.com/-eIw6oya1u_M/Tm5UY8kgqLI/AAAAAAAAAHU/7lb9llwH_mA/h301/11%2B-%2B1" alt="Metawatch charging" title="Metawatch charging" />
<img src="https://lh5.googleusercontent.com/-HU0RYcterXg/Tm5Us5KDW9I/AAAAAAAAAHg/I7OgGYcqrkE/h301/11%2B-%2B1" alt="Usb clip + pins on watch underside" title="Usb clip + pins on watch underside" /></p>
Tarball xz2011-08-31T00:00:00+00:002011-08-31T00:00:00+00:00https://tjtelan.com/blog/tarball-xz/<p>At work I manage a handful of servers running FreeBSD. Today I was writing a little backup script where I wanted the output to be an xz compressed tarball.</p>
<p><a rel="noopener nofollow" target="_blank" href="http://en.wikipedia.org/wiki/Xz">Link to xz Wikipedia article</a></p>
<p>Arch Linux's packages are delivered as tar.xz, and the compression rate is better than gzip and bzip2.</p>
<p>I thought this was going to be pretty straightforward, but I was mistaken. Turns out that the BSD tar implementation of tar does not transparently support xz compression in the same way GNU tar implements it. (I'm not saying that it isn't supported, but -z gzip / -j bzip2 / -J xz is easier to remember).</p>
<p>Being in a particularly untrusting mood towards tar, I figured I would write a portable one-liner that I could use on work servers and on my workstation.</p>
<p>Here goes...</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span>tar -cf - {PATH_TO_ARCHIVE} </span><span style="color:#569cd6;">| </span><span>xz -2ec > {PATH_TO_SAVE_TARBALL}.tar.xz
</span></code></pre>
<p>[<code>-2ec</code> refers to extreme compression level 2, which works for me in this case. Refer to the xz manpage for more info.]</p>
Ubuntu Breakup2011-07-06T00:00:00+00:002011-07-06T00:00:00+00:00https://tjtelan.com/blog/ubuntu-breakup/<p>Summer means I can experiment with my laptop again!</p>
<p>Something I've had on my to-do list for awhile is to try out another Linux distro. In the past I've tried a handful of different ones: starting back in the day with <a rel="noopener nofollow" target="_blank" href="http://www.redhat.com">Red Hat</a>, some live usb trials using <a rel="noopener nofollow" target="_blank" href="http://www.damnsmalllinux.org">DSL</a>, <a rel="noopener nofollow" target="_blank" href="http://www.puppylinux.org">Puppy</a>, maintanence with <a rel="noopener nofollow" target="_blank" href="http://www.sysresccd.org/Main_Page">SystemRescueCd</a> and a couple of attempts at learning projects with <a rel="noopener nofollow" target="_blank" href="http://www.backtrack-linux.org">Backtrack</a>, <a rel="noopener nofollow" target="_blank" href="http://www.linuxfromscratch.org">LFS</a> and <a rel="noopener nofollow" target="_blank" href="http://www.damnvulnerablelinux.org">DVL</a>, a really serious trial with <a rel="noopener nofollow" target="_blank" href="http://www.gentoo.org">Gentoo</a> but always giving up and settling with <a rel="noopener nofollow" target="_blank" href="http://www.ubuntu.com">Ubuntu</a> since it just worked.</p>
<p>However, my general usage has changed significantly from when I started using Ubuntu. Also, I noticed that I've just been resisting basically any upgrades to the OS to the point where I've only been running the LTS versions to guarantee that I wouldn't lose hardware and power saving support after an upgrade (probably a little irrational).</p>
<p>I've been migrating off of the default Ubuntu install for a while now, and I just want a little more control. Starting to sound a little bit like a breakup, but I just want more control while safely distancing myself from Linux beginners.</p>
<p>I picked <a rel="noopener nofollow" target="_blank" href="http://archlinux.org">Arch</a> because of its similarities to BSD. I manage <a rel="noopener nofollow" target="_blank" href="http://www.freebsd.org">FreeBSD</a> servers at work, and I think it is really great. Easy to set up, Ports package management system is amazing, knowledgable (yet kind of stiff) community. However, I don't want to run a FreeBSD desktop, no thanks, sorry.</p>
<p>The learning curve was not as steep as I anticipated. One of the goals I needed to achieve before allowing myself to have a GUI environment was to learn how to use my wifi card in command line. Done and done. To reward myself, I installed <a rel="noopener nofollow" target="_blank" href="http://awesome.naquadah.org">AwesomeWM</a>. I picked this instead of the Xmonad from my previous posts because of the configuration language. I just don't see myself enjoying Haskell over Lua for simple tweaks. If I'm using this computer everyday, I don't want to make it a chore.</p>
<p>Anyway, I'll probably be checking in with some new stuff soon!</p>
Gnome Xmonad Lucid2011-05-27T00:00:00+00:002011-05-27T00:00:00+00:00https://tjtelan.com/blog/gnome-xmonad-lucid/<p>I just got started using tiling window managers because I find that I am a little more productive when I don't use the mouse. With Xmonad, I don't have to deal with overlapping windows and I never have to use the mouse!</p>
<p>I did have some trouble integrating Xmonad into my Ubuntu 10.04 install. I couldn't interface with my wireless card, and I can't always be somewhere with a cable. I had to continue using the NetworkManager app until I had more time to play and look through manuals.</p>
<p>Luckily, I stumbled upon <a rel="noopener nofollow" target="_blank" href="http://markhansen.co.nz/xmonad-ubuntu-lucid">this straightforward walkthrough</a> (Link no longer active). Now I am happily using Gnome and Xmonad in Ubuntu 10.04 Lucid!</p>
Senior Project Update2011-05-27T00:00:00+00:002011-05-27T00:00:00+00:00https://tjtelan.com/blog/senior-project-update/<p>Presentations are just around the corner and I haven't been making quite as many updates with regard to my project.</p>
<p>The quick summary will have to do for now: It is using an ARM based microcontroller (Freescale Kinetis K60 -- tower system). The function it is serving is web-enabled home automation. I'm using the MQX real-time operating system. Users of this system will control the devices connected to the system through a web browser.</p>
<p>The code for this project is up on github, so anyone can contribute, customize for themselves or just look at the <a rel="noopener nofollow" target="_blank" href="https://github.com/tjtelan/homewatchdog">code</a>.</p>
<p>I plan to have a reflection up after next Thursday.</p>
<p><em>Deep breath</em></p>
<p>Still so much to add.</p>
Cygwin Screen Reattach Workaround2011-04-04T00:00:00+00:002011-04-04T00:00:00+00:00https://tjtelan.com/blog/cygwin-screen-reattach-workaround/<p>In an <a href="https://tjtelan.com/blog/virtualbox-headless-mode-on-windows-7/">earlier post</a> I mentioned how I was using a VM for my irc usage. I was running GNU screen + irssi to keep a persistent connection so I can idle in channels.</p>
<p>I've been experimenting with using Cygwin to make my Windows 7 machine my SSH box. So I installed irssi + screen... Looks like everything is all good.</p>
<p>(BTW: If you use a terminal emulator and you haven't looked into GNU screen, I really recommend that you do.)</p>
<p>However, I've been bumping into issues with reattaching. After running both screen <code>-DR</code> and <code>-DRR</code> just hung there.</p>
<p>Quick search for a solution led me to [this workaround][cyg-workaround</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#608b4e;">#!/bin/sh
</span><span style="color:#608b4e;"># Send SIGHUP to each screen to force it to let go and let the server recover
</span><span>ps -as </span><span style="color:#569cd6;">| </span><span>grep screen </span><span style="color:#569cd6;">| </span><span>cut -c4-7 </span><span style="color:#569cd6;">| </span><span>xargs kill -1
</span><span style="color:#608b4e;"># Reattach
</span><span>screen -xRR
</span></code></pre>
<p>This works for me. I'd like to not have to go through all the extra trouble, but it is pretty slick.</p>
Zsh at Win 7 Start up2011-03-28T00:00:00+00:002011-03-28T00:00:00+00:00https://tjtelan.com/blog/zsh-at-win-7-start-up/<p>Beginning of the quarter is tomorrow, and I'm trying to accept that I will be spending more time on a Windows workstation to access the software I'm using for my senior project (more on that soon).</p>
<p>There are a few comforts that I am still missing in Win7 that I'm trying to solve with Cygwin, like <a href="https://tjtelan.com/blog/improving-cygwin-in-windows-with-rxvt/">Improving Cygwin in Windows with Rxvt</a> getting zsh back.</p>
<p>Today, that issue was being able to write Zsh scripts that would run at login through Cygwin instead of using cmd.exe.</p>
<p>Turned out to not be quite as straightforward as I had hoped, but still easy.</p>
<h2 id="add-cygwin-to-win7-path-environment-variable">Add Cygwin to Win7 PATH environment variable.<a class="zola-anchor" href="#add-cygwin-to-win7-path-environment-variable" aria-label="Anchor link for: add-cygwin-to-win7-path-environment-variable">🔗</a></h2>
<p>It took me a moment to find this information after having trouble using <code>ls</code> in my script. Although, <code>echo</code> worked...</p>
<p>Anyway, might as well add this first and avoid the issues entirely.</p>
<ul>
<li>Open start menu</li>
<li>Right-click <code>Computer</code></li>
<li>Click <code>Properties</code></li>
<li>Click <code>Advanced system settings</code></li>
</ul>
<p>In the new <code>System Properties</code> window, click the <code>Environment Variables</code> button located at the bottom.</p>
<p>In the new <code>Environment Variables</code> window, at the bottom are the <code>System variables</code>
Add <code>CYGWIN_HOME</code> with a value of the Cygwin installation path (default is <code>C:/cygwin</code>) if it does not exist in the variable list.</p>
<p>Edit the <code>PATH</code> variable to include <code>%CYGWIN_HOME%/bin</code> which is the same as
including <code>/bin</code> in the Cygwin environment.</p>
<h2 id="run-cygwin-scripts-from-windows">Run Cygwin scripts from Windows<a class="zola-anchor" href="#run-cygwin-scripts-from-windows" aria-label="Anchor link for: run-cygwin-scripts-from-windows">🔗</a></h2>
<p>You can't just write the script and use Zsh built-ins in Windows space. You have to write the script in Zsh space and have windows have Zsh run that script. Simple. Right?</p>
<p>I'll just show my test scripts:</p>
<p><em>(On the Windows desktop)</em></p>
<p>testscript.bat</p>
<pre data-lang="bat" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bat "><code class="language-bat" data-lang="bat"><span>zsh.exe /home/[username]/test.sh
</span></code></pre>
<p><em>(In Cygwin)</em></p>
<p>test.sh</p>
<pre data-lang="sh" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#608b4e;">#!/usr/bin/env zsh
</span><span>echo `pwd`
</span><span>echo Test
</span><span>echo `ls /home/</span><span style="color:#569cd6;">[</span><span>username</span><span style="color:#569cd6;">]</span><span>`
</span></code></pre>
<p>These are the results of running testscript.bat in cmd.exe</p>
<pre data-lang="bat" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bat "><code class="language-bat" data-lang="bat"><span>/cygdrive/d/Users/[username]/Desktop
</span><span>Test
</span><span>test.sh
</span></code></pre>
Virtualbox Headless-mode on Windows 72011-03-21T00:00:00+00:002011-03-21T00:00:00+00:00https://tjtelan.com/blog/virtualbox-headless-mode-on-windows-7/<p>An impromptu reformat of my Windows 7 machine quickly had me frustrated with Virtualbox reconfiguration. It had been so long since I had originally done it, and I had forgotten to document the nuances of that setup. So here goes:</p>
<h2 id="purpose">Purpose<a class="zola-anchor" href="#purpose" aria-label="Anchor link for: purpose">🔗</a></h2>
<p>This specific VM configuration is FreeBSD 8.2 w/ ZFS on root. I don't have the time to set it up from scratch, but I'm currently using the newest ZFS version available on <a rel="noopener nofollow" target="_blank" href="http://mfsbsd.vx.sk">this site</a> (v28, special edition).</p>
<p>I'm only planning on using this VM for SSH for a persistent irc connection. I also want this to start automatically at Win7's boot.</p>
<h2 id="install-vm">Install VM<a class="zola-anchor" href="#install-vm" aria-label="Anchor link for: install-vm">🔗</a></h2>
<p>So after installing FreeBSD and ports and of course, remembering to <em>ENABLE SSH</em>... I had to test the ssh connection.</p>
<p>(Remember to open the ports on the host side. I'm not going to walk you through that step.)</p>
<h2 id="headless-vm">Headless VM<a class="zola-anchor" href="#headless-vm" aria-label="Anchor link for: headless-vm">🔗</a></h2>
<p>Now for the headless start.</p>
<p>I tried creating a .bat file that used <code>VBoxHeadless</code>. This let it start at boot, but it left an annoying cmd.exe window open.</p>
<p>Apparently a common issue. A common solution I used was to create a .vbs script to run the .bat file. This was both annoying and tedious but effective. Startup at boot and no lingering cmd.exe window.</p>
<p>Here are the contents of my .bat and .vbs files. I placed these in the same directory, and made a shortcut in the <code>Start > All Programs > Startup</code> to the <code>.vbs</code> file.</p>
<h3 id="startfreebsdvm-bat">startfreebsdvm.bat<a class="zola-anchor" href="#startfreebsdvm-bat" aria-label="Anchor link for: startfreebsdvm-bat">🔗</a></h3>
<pre data-lang="bat" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bat "><code class="language-bat" data-lang="bat"><span>D:\</span><span style="color:#d69d85;">"Program Files"</span><span>OracleVirtualBoxVBoxHeadless.exe -s FreeBSD startvm-headless.vbs
</span><span style="color:#569cd6;">Set </span><span>WshShell =</span><span style="background-color:#282828;color:#d69d85;"> WScript.CreateObject(</span><span style="color:#d69d85;">"WScript.Shell"</span><span>)
</span><span>obj = WshShell.Run(</span><span style="color:#d69d85;">"D:Users[username]startfreebsdvm.bat"</span><span>, </span><span style="color:#b5cea8;">0</span><span>)
</span><span style="color:#569cd6;">set </span><span>WshShell =</span><span style="background-color:#282828;color:#d69d85;"> Nothing</span><span>
</span></code></pre>
Improving Cywgin in Windows with rxvt2011-03-18T00:00:00+00:002011-03-18T00:00:00+00:00https://tjtelan.com/blog/improving-cygwin-in-windows-with-rxvt/<p>I've grown accustomed to using a terminal emulator pretty much any time I sit down to use the computer.</p>
<p>Most of the time I am using my laptop, which is running Linux (Ubuntu, at the moment). I've already got that workstation set up mostly how I like it. But lately I've been using my Windows 7 desktop a lot more for school related (and gaming related) things.</p>
<p>I use SSH all the time, and when you are using Windows, that means that your choices are limited when looking for a decent SSH client. The only choices I'm going to be considering are Cygwin and PuTTY.</p>
<h2 id="putty">PuTTY<a class="zola-anchor" href="#putty" aria-label="Anchor link for: putty">🔗</a></h2>
<p>PuTTY is alright, but these are the issues I've had with it:</p>
<ul>
<li>Copy/Paste functionality is different.</li>
<li>Highlighting text auto-copies to clipboard. Right click auto-pastes into the console.</li>
<li>No simple way to set up passwordless login using SSH keys</li>
</ul>
<h2 id="cygwin">Cygwin<a class="zola-anchor" href="#cygwin" aria-label="Anchor link for: cygwin">🔗</a></h2>
<p>The learning curve for Cygwin is a little steep, but the issues I've had
with it (without rxvt):</p>
<ul>
<li>Runs <strong>inside</strong> cmd.exe</li>
<li>Cannot resize the window larger than default size without messing with cmd.exe window properties</li>
<li>Copy/Paste is too difficult to use
<ul>
<li>Attempting to highlight text does nothing unless you are in 'mark mode'. Figure that out.</li>
</ul>
</li>
<li>Can't place shortcut in taskbar
<ul>
<li>Because the shortcut to cygwin is to a *.bat rather than *.exe</li>
</ul>
</li>
</ul>
<p><strong>So what is the solution to this mess?</strong></p>
<h1 id="cygwin-rxvt">Cygwin + Rxvt<a class="zola-anchor" href="#cygwin-rxvt" aria-label="Anchor link for: cygwin-rxvt">🔗</a></h1>
<ul>
<li>Install the <em>rxvt</em> package with the cygwin installer</li>
<li>Create shortcut for rxvt</li>
</ul>
<p>You can choose your own colors, fonts, and login shell</p>
<p>Here is the path I used in my shortcut:</p>
<p>cygwin-rxvt-launcher:</p>
<pre data-lang="cmd" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-cmd "><code class="language-cmd" data-lang="cmd"><span>D:\cygwinbinrxvt.exe -sr -sl </span><span style="color:#b5cea8;">2500</span><span> -sb -geometry 90x30 -fg green -bg black -tn rxvt -fn </span><span style="color:#d69d85;">"Anonymous Pro-16"</span><span> -e /usr/bin/zsh --login -i
</span></code></pre>
<p>Here, I'm using green text, black background and my favorite monospaced font, <a rel="noopener nofollow" target="_blank" href="http://www.ms-studio.com/FontSales/anonymouspro.html">Anonymous Pro</a>, at size 16 with Z-shell as my login shell.</p>
<p>Took me a moment to figure out copy/paste, but here is how you do it. It is an improvement, but not perfect:</p>
<p><strong>Copy</strong>:
Same as PuTTY. Highlight using mouse to copy text to the clipboard</p>
<p><strong>Paste</strong>:
Hold down <em>Shift</em> and <em>left-click</em> with your mouse.</p>
About1970-01-01T00:00:00+00:001970-01-01T00:00:00+00:00https://tjtelan.com/about/<p>My name is T.J. Telan. I am a software engineer based in Seattle. This blog is where I host my writing. Occasionally I'll submit to other platforms but I'll always repost here.</p>
<p>I work on at <a rel="noopener nofollow" target="_blank" href="https://www.infinyon.com/">Infinyon</a> on <a rel="noopener nofollow" target="_blank" href="https://www.fluvio.io/">Fluvio</a>. On my free time I hack on <a rel="noopener nofollow" target="_blank" href="https://github.com/orbitalci/orbital">Orbital CI</a>.</p>
<p>I write about:</p>
<ul>
<li>Being productive with new technology </li>
<li>Explanation of complex software systems</li>
<li>Automation</li>
<li>Programming languages</li>
<li>Security</li>
<li>Open Source software</li>
</ul>
<h3 id="i-post-my-blogs-here">I post my blogs here</h3>
<ul>
<li><a rel="noopener nofollow" target="_blank" href="https://dev.to/tjtelan">Dev.to</a>.</li>
<li><a rel="noopener nofollow" target="_blank" href="https://medium.com/@tjtelan">Medium</a></li>
</ul>
<h3 id="newsletter">Newsletter!</h3>
<p><em>If you want to get an email whenever I post a new blog then subscribe to my newsletter.</em></p>
<!-- In the about.html template, I have the newsletter form included after here. -->