\documentclass[12pt]{article}
\usepackage{fullpage}
\usepackage{amsmath,amsopn}
\usepackage{graphicx}
\usepackage{color}
\usepackage{verbatim}
\usepackage{setspace}
\usepackage{gensymb}
\usepackage{float}
\usepackage[normalem]{ulem}
%\usepackage[dvips,bookmarks=false,colorlinks,urlcolor=blue,pdftitle={
\usepackage{caption}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{amsthm}
\usepackage{mathdefs}

\def\bx{{\bf x}}
\def\by{{\bf y}}
\def\bz{{\bf z}}
\def\V{{\bf V}}
\def\vl{{\bf l}}
\def\vm{{\bf m}}
\def\mS{{\bf S}}
\def\mT{{\bf T}}
\def\bH{{\bf H}}
\def\mA{{\bf A}}
\def\mM{{\bf M}}
\def\mP{{\bf P}}
\def\tx{{\tilde{\bf x}}}
\def\ta{{\tilde{\bf a}}}
\def\tb{{\tilde{\bf b}}}
\def\tc{{\tilde{\bf c}}}
\def\hn{{\bf \hat{n}}}
\def\hv{{\bf \hat{v}}}
\def\hh{{\bf \hat{h}}}
\def\vh{{\bf h}}
\def\vs{{\bf s}}
\def\hs{{\bf \hat{s}}}
\newcommand{\Ev}{\mathbf{E}}
\newcommand{\Fv}{\mathbf{F}}

\usepackage[bookmarks=false,colorlinks,urlcolor=blue,pdftitle={Take-home Quiz 9},pdfauthor={Matthew O'Toole}]{hyperref}

\begin{document}

\begin{centering}
	{\large 16-385 Computer Vision, Fall 2020\\}
	\vspace{2ex}
	{\LARGE Take-home Quiz 9\\}
	\vspace{2ex}
	{\large Due Date: Monday November 16, 2020 23:59\\}
\end{centering}

\bigskip

\section*{Question 1}

As we discussed in class, neural networks use non-linear activation functions to connect their hidden layers. 

\begin{enumerate}
\item Prove that if we have a \emph{linear} activation function, then the number of hidden layers has no effect on the actual network.

\item A common activation function is the sigmoid function $\sigma(x) = \frac{1}{1 + e^{-x}}$. Derive the gradient of the sigmoid function with respect to $x$, and show that it can be written as a function of $\sigma(x)$ itself---that is, if we know the value of the sigmoid, we can also compute its gradient without having access to $x$ directly.  \emph{Hint: We (very) briefly saw the gradient of the sigmoid function in class.}

\item How does the gradient of the sigmoid activation function behaves as the absolute value of $x$ increases? Can you think of any problems this behavior may create for the gradient descent algorithm, when the sigmoid is used as the activation function for many layers?

\item Often, the sigmoid function is replaced with the hyperbolic tangent function, $\tanh(x) = \frac{1-e^{-2x}}{1+e^{-2x}}$. Show that this function and its gradient can be written in terms of  $\sigma(x)$.

\item What are the output ranges of the sigmoid and the hyperbolic tangent functions? When would we prefer to use each of these functions?
\end{enumerate}

\section*{Instructions}

\begin{enumerate}
    \item \textbf{Integrity and collaboration:} Students are encouraged to work
    in groups but each student must submit their own work. If you work as a
    group, include the names of your collaborators in your write up. Plagiarism
    is strongly prohibited and may lead to failure of this course.

    \item \textbf{Questions:} If you have any questions, please look at Piazza
    first. Other students may have encountered the same problem, and it may be solved
    already. If not, post your question on the discussion board. Teaching staff will
    respond as soon as possible.

    \item \textbf{Write-up:} Your write-up should be typeset in \LaTeX and should consist of your answers to the theory questions. Please note that we
    \textbf{\color{red}do not} accept handwritten scans for your write-up in quizzes.

    \item \textbf{Submission:} Your submission for this take-home quiz should be a
    PDF file, \texttt{<andrew-id.pdf>}, with your write-up. \textbf{\color{red}Please do not submit ZIP files.}
\end{enumerate}

\end{document}
